For example, if the property "excludeJCC=2.0" is specified in "<myTest>_app.properties", the harness will skip "myTest" for any versions of JCC at or earlier than 2.0.
Attached is a patch that enhances this "excludeJCC" property in two ways:
1) Instead of only allowing exclusion of versions at or earlier than the specified version, this patch adds support for two new keywords, "at-or-before" and "at-or-after", to allow more control of which versions of JCC are excluded. If neither of these keywords is given, the behavior defaults to the old behavior--i.e. the harness will exclude all versions at or earlier than the specified version.
For example, if the property "excludeJCC=at-or-after:2.0" is specified in "<myTest>_app.properties", the harness will skip "myTest" for any versions of JCC at or _later_ than 2.0.
2) Adds a new JVM clause for further narrowing the exclusion criteria. This clause starts with the ",when" tag and is followed by a three-part jvm version. In this case, a test will only be skipped if BOTH the JCC clause AND the jvm clause are true. For example:
To skip a test when running with any version of the IBM Universal Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1
To skip a test when running with any version of the IBM Universal Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or later:
excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1
The patch for these two changes is attached to this email. It includes the addition of a short description of these changes in the java/testing/README.htm file. I've also attached that file (README.htm) as a regular attachment so that the changes are more easily viewable by any reviewers.
I've tried to make these changes in a modular way so that if a similar property is ever created for the new Derby Client, this same code can be used.
svn stat:
M java\testing\README.htm
M java\testing\org\apache\derbyTesting\functionTests\harness\RunList.java
M java\testing\org\apache\derbyTesting\functionTests\harness\RunTest.java
M java\testing\org\apache\derbyTesting\functionTests\suites\derbynetmats.properties
I've run the complete "derbyall" suite twice with this patch (using different versions of JCC and different JVMs) on Windows, and all tests ran/passed as expected.
Army
Index: java/testing/README.htm
===================================================================
--- java/testing/README.htm (revision 164284)
+++ java/testing/README.htm (working copy)
@@ -424,7 +424,8 @@
</ul>
<li>i18nTest</li>
<ul>
- <li>tests that characters outside simple ascii scope do not result in
errors.
+ <li>tests that characters outside simple ascii scope do not result
+in errors. </li>
<li>takes 5 to 10 minutes<br>
</li>
</ul>
@@ -1065,8 +1066,38 @@
The skip setting does not go into the subversion level, i.e. setting
runwithjdk141=false has no effect, and setting runwithjdk14 affects
runs with jdk141 as well as jdk142.<br>
-Other skip reasons are encryption protocols specific to a certain jvm. <br>
+Other skip reasons are encryption protocols specific to a certain
+jvm.<br>
</p>
+<p>The property for skipping a test based on the version of the IBM
+Universal Driver is "excludeJCC". The keywords "<span
+ style="font-weight: bold;">at-or-before</span>" and "<span
+ style="font-weight: bold;">at-or-after</span>" can be used to specify
+which range of JCC versions should be excluded. If neither of
+these keywords is provided, the default is "<span
+ style="font-weight: bold;">at-or-before</span>". For example:<br>
+</p>
+To skip a test when running with any version of the IBM Universal
+Driver that is 2.4 or earlier:<br>
+excludeJCC=at-or-before:2.4<br>
+<br>
+To skip a test when running with any version of the IBM Universal
+Driver that is 2.0 or later:<br>
+excludeJCC=at-or-after:2.0<br>
+</>
+<p>You can also specify an (optional) jvm clause to further tune the
+exclusion criteria. This clause starts with the "<span
+ style="font-weight: bold;">,when</span>" tag and is followed by a
+three-part jvm version. In this case, a test will only be skipped
+if BOTH the JCC clause AND the jvm clause are true. For example:<br>
+</p>
+<p>To skip a test when running with any version of the IBM Universal
+Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:<br>
+excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1<br>
+</p>
+To skip a test when running with any version of the IBM Universal
+Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or later:<br>
+excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1<br>
<p>Another skipping mechanism works on entire 'frameworks'. Currently
there are only 3 supported in the harness, embedded, NetworkServer with
the IBM Universal JDBC Driver ('DerbyNet') and NetworkServer with the
@@ -1123,7 +1154,11 @@
sets additional properties.<br>
java
-D=TestSpecialFlags=derby.infolog.append=true
-org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql</td>
+org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql<br>
+excludeJCC<br>
+ See above section <a
+ href="#skipping">4.10</a><br>
+ </td>
</tr>
</tbody>
</table>
Index: java/testing/org/apache/derbyTesting/functionTests/harness/RunList.java
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/harness/RunList.java
(revision 164284)
+++ java/testing/org/apache/derbyTesting/functionTests/harness/RunList.java
(working copy)
@@ -62,6 +62,7 @@
static boolean needJdk12 = false;
static boolean needJdk12ext = false;
static boolean excludedFromJCC = false;
+ static String clientExclusionMessage;
static Boolean needIBMjvm = null;
static boolean needJdk14 = false;
static boolean needEncryption = false;
@@ -118,6 +119,11 @@
static Properties specialProperties; // for testSpecialProps
static BufferedReader runlistFile;
+ static String [] clientExclusionKeywords = new String [] {
+ "at-or-before:", "at-or-after:", "when-at-or-before:jdk",
+ "when-at-or-after:jdk", "when:jdk"
+ };
+
public RunList()
{
}
@@ -270,8 +276,8 @@
else if(needJdk14)
pwOut.println("Cannot run the suite, requires jdk14 or
higher, have jdk" + javaVersion);
else if(excludedFromJCC)
- pwOut.println("Cannot run the suite on JCC version " +
excludeJCC + " or lower.");
- else if((needIBMjvm == null ||
needIBMjvm.booleanValue() == false))
+ pwOut.println(clientExclusionMessage);
+ else if((needIBMjvm == null ||
needIBMjvm.booleanValue() == false))
pwOut.println("Cannot run the suite, requires IBM jvm,
jvm vendor is " + System.getProperty("java.vendor"));
else
pwOut.println("Cannot run the suite, have jdk" +
javaVersion);
@@ -1115,25 +1121,14 @@
if (verbose) System.out.println("Exception in shouldSkipTest: "
+ e);
}
- if (excludeJCC != null) {
- int excludeMajor = 0;
- int excludeMinor = 0;
- try
- {
- excludeMajor =
Integer.parseInt(excludeJCC.substring(0,excludeJCC.indexOf(".")));
- excludeMinor =
Integer.parseInt(excludeJCC.substring(excludeJCC.indexOf(".")+1));
- } catch (NumberFormatException nfe) {
- System.out.println("excludeJCC property poorly formatted: "
+ excludeJCC);
- } catch (NullPointerException npe) {
- System.out.println("excludeJCC property poorly formatted: "
+ excludeJCC);
+ try {
+ checkClientExclusion(excludeJCC, "JCC", jccMajor,
jccMinor, javaVersion);
+ } catch (Exception e) {
+ excludedFromJCC = true;
+ clientExclusionMessage = e.getMessage();
+ return true;
}
- if (excludeMajor >= jccMajor && excludeMinor >= jccMinor)
- {
- excludedFromJCC = true;
- return true;
- }
- }
- }
+ }
return result; // last test result is returned
}
@@ -1240,5 +1235,309 @@
ps.flush();
}
+ /* ****
+ * Look at the received exclusion property and use it to
+ * figure out if this test/suite should be skipped based
+ * on the actual client and JVM versions in question.
+ * @param exclusion The harness property indicating the
+ * rules for skipping this test. For example:
+ * "at-or-before:2.0,when-at-or-after:jdk1.5.1".
+ * @param clientName Name of the client being used.
+ * @param clientMajor The 'major' part of the client version
+ * that is actually being used for the test.
+ * @param clientMinor The 'minor' part of the client version
+ * that is actually being used for the test.
+ * @param javaVersion JVM being used, as a string.
+ * @return Exception is thrown if this test/suite should
+ * be skipped; else we simply return.
+ */
+ public static void checkClientExclusion(String exclusion,
+ String clientName, int clientMajor, int clientMinor,
+ String javaVersion) throws Exception
+ {
+
+ if (exclusion == null)
+ // we already know the test isn't excluded.
+ return;
+
+ // These tell us whether we want to 1) exclude version
+ // numbers that are lower than the target version, or
+ // 2) exclude version numbers that are higher than the
+ // target version.
+ int clientComparisonType = 0;
+ int jvmComparisonType = 0;
+
+ exclusion = exclusion.toLowerCase();
+ String clientVersion = null;
+
+ // Figure out what kind of comparison we need for the client version.
+ int comma = exclusion.indexOf(",");
+ if (comma != -1)
+ clientVersion = exclusion.substring(0, comma);
+ else
+ clientVersion = exclusion;
+
+ try {
+ clientComparisonType = getVersionCompareType(clientVersion);
+ } catch (Exception e) {
+ System.out.println("exclusion property poorly formatted: " +
exclusion);
+ return;
+ }
+
+ // Figure out what kind of comparison we need for the JVM version.
+ boolean jvmDependent;
+ if (comma == -1)
+ jvmDependent = false;
+ else {
+ jvmDependent = true;
+ // "+6" in next line is length of ",when-", which is the
+ // keyword used to begin the jvm exclusion string.
+ String jvmVersion = exclusion.substring(comma+6);
+ try {
+ jvmComparisonType = getVersionCompareType(jvmVersion);
+ } catch (Exception e) {
+ System.out.println("exclusion property poorly formatted: " +
exclusion);
+ return;
+ }
+ }
+
+ // Load the client and JVM target versions. The "5" in the
+ // next line means that we want to parse out 5 numbers from
+ // the property: 2 numbers for the client version (ex. "2.0")
+ // and 3 numbers for the JVM version (ex. "1.5.1").
+ int [] excludeInfo = null;
+ try {
+ excludeInfo = getVersionArray(exclusion, 5);
+ } catch (Exception e) {
+ System.out.println("Unexpected text in exclusion property: " +
e.getMessage());
+ return;
+ }
+
+ // Now check to see if this test/suite should be excluded.
+ // First check the client version.
+ if (versionExcluded(new int [] {clientMajor, clientMinor}, 0,
+ excludeInfo, 0, 2, clientComparisonType))
+ {
+
+ if (!jvmDependent) {
+ // then skip it regardless of JVM.
+ throw new Exception("This test/suite is excluded from running
with " +
+ clientName + " versions at or " +
+ (clientComparisonType == -1 ? "before " : "after ") +
+ excludeInfo[0] + "." + excludeInfo[1] + ".");
+ }
+
+ // Now check the JVM version.
+ int [] jvmInfo = null;
+ try {
+ jvmInfo = getVersionArray(javaVersion, 3);
+ } catch (Exception e) {
+ System.out.println("Unexpected text in exclusion property: " +
e.getMessage());
+ return;
+ }
+
+ if (versionExcluded(jvmInfo, 0, excludeInfo, 2, 3,
jvmComparisonType)) {
+ throw new Exception("This test/suite is excluded from running
with " +
+ clientName + " versions at or " +
+ (clientComparisonType == -1 ? "before " : "after ") +
+ excludeInfo[0] + "." + excludeInfo[1] + " when JVM
versions at or " +
+ (jvmComparisonType == -1 ? "before " : "after ") +
+ excludeInfo[2] + "." + excludeInfo[3] + "." +
excludeInfo[4] +
+ " are being used.");
+ }
+ }
+
+ }
+
+ /* ****
+ * Parses a versionString property and returns the specified
+ * number of integers as found in that string. If the number
+ * of integers requested is larger than the number of integers
+ * found in the version string, -1 will be used as filler.
+ *
+ * An example versionString might be any of the following:
+ * "2.4" or "at-or-after:2.4" or "when:jdk1.3.1" or
+ * "when-at-or-after:jdk1.3.1", etc. In these examples,
+ * the resultant int arrays would be:
+ *
+ * "2.4" ==> [2,4] // if resultSize == 2.
+ * "at-or-after:2.4" ==> [2.4] // if resultSize == 2.
+ * "when:jdk1.3.1" ==> [1,3,1] // if resultSize == 3.
+ * "when-at-or-after:jdk1.3.1" ==> [1,3,1,-1] // if resultSize == 4.
+ *
+ * @param versionString The version string to parse.
+ * @param resultSize The number of integers to parse out of the
+ * received version string.
+ * @return An integer array holding resultSize integers as parsed
+ * from the version string (with -1 as a filler if needed).
+ */
+ private static int [] getVersionArray(String versionString, int resultSize)
+ throws Exception
+ {
+
+ if (versionString == null)
+ // Use an empty string so that tokenizer will still work;
+ // result will be an array of "-1" values.
+ versionString = "";
+
+ int [] result = new int[resultSize];
+
+ String tok = null;
+ String text = null;
+ StringTokenizer st = new StringTokenizer(versionString, ".,_");
+ for (int i = 0; i < resultSize; i++) {
+
+ if (!st.hasMoreTokens()) {
+ // if we're out of integers, use -1 as a filler.
+ result[i] = -1;
+ continue;
+ }
+
+ // Get the token and parse out an integer.
+ tok = st.nextToken();
+ int pos = 0;
+ for (; !Character.isDigit(tok.charAt(pos)); pos++);
+ text = tok.substring(0, pos);
+
+ // If we have text, make sure it's a valid keyword
+ // and then move past it.
+ if ((text.length() > 0) && !isClientExclusionKeyword(text))
+ throw new Exception(text);
+
+ // Load the int.
+ tok = tok.substring(pos);
+ if (tok.length() == 0) {
+ // no integer found, so don't count this iteration.
+ i--;
+ continue;
+ }
+
+ result[i] = Integer.parseInt(tok);
+
+ }
+
+ return result;
+
+ }
+
+ /* ****
+ * Looks at a version string and searches for an indication
+ * of what kind of versions (lower or higher) need to be
+ * excluded. This method just looks for the keywords
+ * "at-or-before" and "at-or-after", and then returns
+ * a corresponding value. If neither of those keywords
+ * is found, the default is to exclude versions that are
+ * lower (i.e. "at-or-before").
+ * @param versionString The version string in question,
+ * for example "2.4" or "jdk1.3.1" or "at-or-before:jdk1.3.1".
+ * @return -1 if we want to exclude versions that come
+ * before the target, 1 if we want to exclude versions
+ * that come after the target. Default is -1.
+ */
+ private static int getVersionCompareType(String versionString)
+ throws Exception
+ {
+
+ if (versionString == null)
+ // just return the default.
+ return -1;
+
+ int colon = versionString.indexOf(":");
+ if (colon != -1) {
+ if (versionString.startsWith("at-or-before"))
+ return -1;
+ else if (versionString.startsWith("at-or-after"))
+ return 1;
+ else
+ throw new Exception("bad exclusion property format");
+ }
+
+ return -1;
+
+ }
+
+ /* ****
+ * Takes two versions, each of which is an array of integers,
+ * and determines whether or not the first (actual) version
+ * should be excluded from running based on the second (target)
+ * version and on the received comparisonType.
+ *
+ * For example, let vActual be [2,1] and vTarget be [2,4]. Then
+ * if comparisonType indicates that versions "at or before" the
+ * the target version (2.4) should be excluded, this method
+ * would return true (because 2.1 is before 2.4); if comparisonType
+ * indicates that versions "at or after" the target type should
+ * be excluded, this method would return false (because 2.1 is
+ * NOT at or after 2.4).
+ *
+ * @param vActual The actual version, as an int array.
+ * @param vTarget The target version, as an int array.
+ * @param offset1 Offset into vActual at which to start the
+ * comparison.
+ * @param offset2 Offset into vTarget at which to start the
+ * comparison.
+ * @param numParts The maximum number of integer parts to compare.
+ * @param comparisonType -1 if we want to exclude versions
+ * at or before the target; 1 if we want to exclude versions
+ * at or after the target.
+ * @return True if the actual version should be excluded from
+ * running, false otherwise.
+ */
+ private static boolean versionExcluded(int [] vActual, int offset1,
+ int [] vTarget, int offset2, int numParts, int comparisonType)
+ {
+
+ // Figure out how many integer parts we can actually compare.
+ // The max is "len", but if len is greater than the length of
+ // either of the versions, then we have to compensate for
+ // the shortest version.
+ int compareLen = (vActual.length >= vTarget.length ? vTarget.length :
vActual.length);
+ compareLen = (compareLen <= numParts ? compareLen : numParts);
+
+ // Now do the comparison.
+ for (int i = 0; i < compareLen; i++) {
+
+ if (comparisonType * vActual[offset1] > comparisonType *
vTarget[offset2])
+ return true;
+
+ if (comparisonType * vActual[offset1] < comparisonType *
vTarget[offset2])
+ return false;
+
+ offset1++;
+ offset2++;
+
+ }
+
+ // If we get here, the two versions are the same thru
+ // compareLen parts. If that's as far as we're supposed
+ // to compare, then we treat them as equal. Else, we take
+ // the version having more parts as the greater of the two.
+
+ if (compareLen == numParts)
+ // treat them as equal.
+ return true;
+
+ return (comparisonType * vActual.length > comparisonType *
vTarget.length);
+
+ }
+
+ /* ****
+ * Checks to see if the received string is a recognized
+ * keyword for an exclusion property.
+ * @param text The string in question.
+ * @return True if the received text is a valid keyword
+ * for exclusion properties; false otherwise.
+ */
+ private static boolean isClientExclusionKeyword(String text) {
+
+ for (int i = 0; i < clientExclusionKeywords.length; i++) {
+ if (clientExclusionKeywords[i].equals(text))
+ return true;
+ }
+
+ return false;
+
+ }
+
}
Index: java/testing/org/apache/derbyTesting/functionTests/harness/RunTest.java
===================================================================
--- java/testing/org/apache/derbyTesting/functionTests/harness/RunTest.java
(revision 164284)
+++ java/testing/org/apache/derbyTesting/functionTests/harness/RunTest.java
(working copy)
@@ -929,7 +929,7 @@
upgradetest = true;
}
- if ( framework.startsWith("DB2") && (!
jvmName.equals("j9_foundation")))
+ if ( framework.startsWith("Derby") && (!
jvmName.equals("j9_foundation")))
{
Class c = null;
@@ -949,22 +949,11 @@
} catch (ClassNotFoundException e) {}
String excludeJcc = sp.getProperty("excludeJCC");
- if (excludeJcc != null) {
- int excludeMajor = 0;
- int excludeMinor = 0;
- try {
- excludeMajor =
Integer.parseInt(excludeJcc.substring(0,excludeJcc.indexOf(".")));
- excludeMinor =
Integer.parseInt(excludeJcc.substring(excludeJcc.indexOf(".")+1));
- } catch (NumberFormatException nfe) {
- System.out.println("excludeJCC property
poorly formatted: " + excludeJcc);
- } catch (NullPointerException npe) {
- System.out.println("excludeJCC property
poorly formatted: " + excludeJcc);
- }
- if (excludeMajor >= jccMajor && excludeMinor >=
jccMinor)
- {
- skiptest = true;
- addSkiptestReason("This test is excluded from
running with JCC versions " + excludeJcc + " and below");
- }
+ try {
+ RunList.checkClientExclusion(excludeJcc, "JCC",
jccMajor, jccMinor, javaVersion);
+ } catch (Exception e) {
+ skiptest = true;
+ addSkiptestReason(e.getMessage());
}
}
@@ -1451,28 +1440,16 @@
jvmnet = true;
String excludeJcc = ap.getProperty("excludeJCC");
- if ( framework.startsWith("DB2") )
+ if ( framework.startsWith("Derby") )
{
- if (excludeJcc != null) {
- int excludeMajor = 0;
- int excludeMinor = 0;
- try {
- excludeMajor =
Integer.parseInt(excludeJcc.substring(0,excludeJcc.indexOf(".")));
- excludeMinor =
Integer.parseInt(excludeJcc.substring(excludeJcc.indexOf(".")+1));
- } catch (NumberFormatException nfe) {
- System.out.println("excludeJCC property poorly
formatted: " + excludeJcc);
- } catch (NullPointerException npe) {
- System.out.println("excludeJCC property poorly
formatted: " + excludeJcc);
- }
- if (excludeMajor >= jccMajor && excludeMinor >= jccMinor)
- {
- skiptest = true;
- addSkiptestReason("This test is excluded from running
with JCC versions " + excludeJcc + " and below");
- }
+ try {
+ RunList.checkClientExclusion(excludeJcc, "JCC", jccMajor,
jccMinor, javaVersion);
+ } catch (Exception e) {
+ skiptest = true;
+ addSkiptestReason(e.getMessage());
}
}
-
// for now we want just want to have a single property
// for all j9 versions
String testJVM = (jvmName.startsWith("j9") ? "j9" : jvmName);
Index:
java/testing/org/apache/derbyTesting/functionTests/suites/derbynetmats.properties
===================================================================
---
java/testing/org/apache/derbyTesting/functionTests/suites/derbynetmats.properties
(revision 164284)
+++
java/testing/org/apache/derbyTesting/functionTests/suites/derbynetmats.properties
(working copy)
@@ -3,3 +3,4 @@
jdk12test=true
runwithj9=false
timeout=60
+excludeJCC=at-or-after:3.0,when-at-or-before:jdk1.3.1
Title: readme.htmDerby Functional Tests
Package: org.apache.derbyTesting
created by [EMAIL PROTECTED]
last updated on 12/02/2004 by: [EMAIL PROTECTED]
- 1. Introduction
- 2. Quickstart
- 2.1 running tests
- 2.2 building
derbyTesting package
- 3. More details on running the derby functional tests
- 3.1 Running 1 test
- 3.2 Running suites of tests
- 4. Harness internals for developers
- 4.1 Test types
- 4.2 Supporting files for tests
- 4.3 <testname>_app.properties
- 4.4 <testname>_derby.properties
- 4.5 tmp files, out files, master files, and canons
- 4.6 Masking and comparing
- 4.7 Adding a new test
- 4.8 Suites and adding a new suite
- 4.9 Running with a new jvm
- 4.10 Skipping a test
- 4.11 Frameworks
- 4.12 Some test harness properties
1. Introduction
This document describes functionality of the derby functional testing package org.apache.derbyTesting. This package is based on the functional tests in use at IBM for testing the Cloudscape product before its contribution to ASF.
In the following, instructions are geared towards a unix
environment. For other environments, some details may need to be
adjusted. For instance, the document may
refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.
In the following the top
directory under which the subversion tree is placed is referred to as
${derby.source} - see also the
derby BUILDING.txt.
The version of the classes and supporting files of the derbyTesting
package have to match the version of the classes of the derby package.
Thus you either need to build all jars yourself, or get all jar files
from the incubator site at the same time when available.
2. QuickStart
2.1 running tests
The derbyTesting package enables you to run 1 test or a suite of tests.
Before you can run, you need to setup your environment:
- Obtain a jdk or jre (based on jdk 1.3.1 specification or higher).
Add the bin directory to your $PATH. Currently supported are:
| jdk131
- Sun
HotSpot jdk1.3.1 jdk141 - Sun HotSpot jdk1.4.1 jdk142 - Sun HotSpot jdk1.4.2 jdk15 - Sun HotSpot jdk1.5 ibm131 - IBM Classic jdk1.3.1 ibm141 - IBM Classic jdk1.4.1 ibm142 - IBM Classic jdk1.4.2 j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition) |
- set $CLASSPATH to include the following jars:
- jakarta-oro-2.0.8.jar oromatcher, obtain from http://jakarta.apache.org/oro/index.html, or follow this link for zip file, or tar.gz)
- derbyTesting.jar
- derby.jar
main derby package classes - derbytools.jar
derby tools classes for tools like ij
and dblook - derbynet.jar
derby network server classes - db2jcc.jar
and db2jcc_license_c.jar
IBM Universal JDBC Driver classes. (See
IBM developerworks
for download) - derbyLocale_*.jar locale files holding translated messages.
-
test files and classes
For example:
| (note that $jardir is
only a convenience variable and that the command below has carriage
returns for formatting reasons): set jardir=/local/derbyjar set CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/db2jcc.jar: $jardir/db2jcc_license_c.jar:$jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar: $jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar: $jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar: $jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar: $CLASSPATH set PATH=/local/jdk141/bin:$PATH |
To run 1 test:
| syntax: java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunTest <testdir>/<testname> where
to run the test supersimple against the embedded driver: java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql To run a test with network server, add -Dframework=DerbyNet to the run. The test harness will to start network server at port 1527 or connect to a running one, run the test, and stop network server thereafter. for example: java -Dframework=DerbyNet org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql |
A successful run will have a .pass file, and the output to the
console will show no difference between expected and actual test
result. A failed test run will have at least a .fail file and the
output to the console will show the difference between expected and
actual result.
To run a suite:
| syntax: java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunSuite <testsuite> where
java -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunSuite derbylang |
Each suite run should be started in a clean directory. The test output directory will not be emptied out before testing is begun, although individual test files and result files will be cleaned out and overwritten.
The suites provided are:
- derbylang:
- basic functionality of
language implementation in derby.
- Mostly .sql type tests.
- tested on a variety of hardware takes from 1.15m to 2.00 hours
- derbynetmats
- basic network server tests.
- variety of tests, including some from derbylang suite
- tested on a variety of hardware takes from 15 to 30 minutes
- derbynetautostart
- tests networkserver functionality without requiring
networkserver framework
- propertyinfo
- runs test to get property information
- storeall
- tests for storage area
- includes:
- storemats: most basic quick verification tests.
- storemore: more extensive storage tests
- storetests: set of store tests grouped together because they do not each need to create a new database
- tested on a variety of hardware takes from 25 to 50 minutes
- xa
- tests the xa implementation. There is both a storage and language element to these tests
- tested on a variety of hardware takes from 2 to 4 minutes
- storeunit
- tests store-related unit tests. Runs from 8 to 15 minutes
- unit
- tests 4 general functionality unit tests. runs from 5 to 10
minutes
- jdbcapi
- tests implementation of jdbc api such as Connection class implementation, Metadata etc.
- takes from 20 to 40 minutes
- jdbc20
- tests implementation of features from the jdbc 20 specification
- takes 2 to 5 minutes
- jdk14
- tests implementation of features from the jdk14 specification
- takes 2 to 5 minutes
- demo, simpledemo
- tests the SimpleApp example
- simpledemo runs SimpleApp itself - and thus has a different default resource package name (namely, no package) than all the other tests. Hence it needed its own suite.properties file.
- takes 30 to 1 minute
- nist
- test obtained from the NIST SQL suite v 6.0
- takes 5 to 10 minutes
- encryptionAll
- takes 30 to 55 minutes
- runs a few encryption tests plus the following encryption tests
suites
- encryption
- runs the storemats, sysinfo and multi suites in encryption
scheme DESede
- takes 25 to 40 minutes
- encryptionAES - tests AES encryption scheme
- encryptionBlowfish - tests Blowfish encryption scheme
- encryptionCFB - tests CFB encryption scheme
- encryptionDES - tests DES encryption scheme
- encryptionECB - tests ECB encryption scheme
- encryptionOFB - tests OFB encryption scheme
- multi
- runs a simple test case with 10 threads
- runs for 10 minutes, then shuts down all threads
- derbytools
- tests for dblook, ij, and import/export utilities
- takes 5 to 10 minutes
- i18nTest
- tests that characters outside simple ascii scope do not result in errors.
- takes 5 to 10 minutes
- derbyall
- contains all suites typically run by all developers
- tested on a variety of hardware takes from 3.00 - 6.00 hours
- See Note1
A successful run with all tests passing will have no *.fail files created, the <testsuite>_fail.txt file will be empty, and the <testsuite>_report.txt file will show no failures in the Summary results section.
| -------snippet from
derbylang_report.txt ----- ----------------------------------------------------------- Summary results: Test Run Started: 2004-11-10 11:27:55.0 Test Run Duration: 00:04:09 129 Tests Run 100% Pass (129 tests passed) 0% Fail (0 tests failed) 0 Suites skipped |
2.2 building
derbyTesting package
To build the derbyTesting package:
- follow all the steps in the derby BUILDING.txt.
This is some typical
output for the ant build process.
| > cd
/local/derby/java/testing > ant.ksh Searching for build.xml ... Buildfile: /local/derby/java/testing/build.xml compile: [javac] Compiling 30 source files to /local/derby/classes ... [copy] Copying 1 file to /local/derby/classes/org/apache/derbyTesting/funct ionTests BUILD SUCCESSFUL Total time: 10 minutes 3 seconds |
Once you have built the derbyTesting package built, you can make a derbyTesting.jar use the jar build target at the ${derby.source}level.
This will look something like:
| c:> ant derbytestingjar Searching for build.xml ... Buildfile: C:\derby\build.xml initjars: [mkdir] Created dir: C:\derby\jars\ [mkdir] Created dir: C:\derby\jars\lists [echo] Revision number set to exported [echo] . derbytestingjar: [echo] Beginning derbytesting.jar build ..... BUILD SUCCESSFULL |
3. More details on running the derby functional tests
The functional tests are run using a class called 'RunTest'. This class
calls a number of other classes. A group of tests, called a 'suite' is
executed using a class called 'RunSuite'.
3.1 Running 1 test
See section 2.1 for the basic steps to run 1 test.
To pass on system level properties to the test harness, use the test harness property -DtestSpecialFlags. For example, to run a test forcing the message text to be retrieved from the network server:
| java -Dframework=DerbyNet -DtestSpecialFlags=RetrieveMessagesFromServerOnGetMessage=true org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql |
Tests will be executed in the current directory. When
running a test
using the network server, i.e. -Dframework=DerbyNet, the test will run
in a subdirectory (automatically created) 'DerbyNet'.
See Note2.
The test will normally create the following:
- a database. The default name is 'wombat'. However, the name may be different depending on certain properties passed in to the test harness.
- a .out file: the final result file
- a .tmp file; the initial result file, before any modification to prevent irrelevant differences has been applied (before 'masking').
- a .diff file; the differences between the .out and the master file with expected output it is compared to.
- a .pass or .fail file. This file lists the test if it passes under .pass, and under .fail if the output in .out is different from the expected output in the master.
possibly created:
- additional files used in a specific test may get copied over to the test directory. These normally do not get cleaned up.
- .tmpstr file is created for network server tests and is a possibly massaged copy of the master file the output needs to be compared with.
- .err and .out files in network server database files for any additional error output.
When the test is successful, cleanup will occur unless the test harness
property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all
files except for .pass.
See Note3.
A successful run (this example is from a dos environment) would look for instance like:
| c:>
derbyTesting.functionTests.harness.RunTest lang/supersimple.sql C:\derby\run2 supersimple -- listing properties -- derby.locks.deadlockTimeout=3 derby.locks.waitTimeout=3 *** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 *** The test should be running... MasterFileName = master/supersimple.out *** End: supersimple jdk1.4.2_03 2004-11-10 16:51:25 *** |
A Test Failure shows the diff, creates a .fail file, does not create
a .pass file, and does not cleanup any files upon completion. The
output might look like this:
| c:>
derbyTesting.functionTests.harness.RunTest lang/supersimple.sql C:\derby\run2 supersimple -- listing properties -- derby.locks.deadlockTimeout=3 derby.locks.waitTimeout=3 *** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 *** The test should be running... MasterFileName = master/supersimple.out 10 del < 10 10a10 > 1 Test Failed. *** End: supersimple jdk1.4.2_03 2004-11-10 16:55:02 *** |
3.2 Running a suite of tests
See section 2.1 for a basic explanation on how to run a suite of tests.
Tests will be run in a subdirectory with the name of the test suite under the current directory. Eg. for derbylang suite, a directory derbylang will be created. While the tests are run, information about the run is inserted into a <testsuite>.sum file. When all tests have completed summary files are created <testsuite>_pass.txt, _fail.txt, and _diff.txt files are created as well as a <testsuite>_report.txt with additional details. Some of the information is duplicate. Also, a .skip file will be created holding a list of the tests that were skipped (for more details on this, see the section on skipping tests).
RunSuite does not empty the top level directory before running. Thus, if another suite was run in the same directory at an earlier time, the resulting summary files might contain results for more than the current run. Therefore it is important to run each suite in a clean directory.
Sample output from RunSuite:
| c:> $ java
org.apache.derbyTesting.functionTests.harness.RunSuite derbylang Top suite: derbylang Suite to run: derbylang:derbylang Now do RunList Now run the suite's tests Run the tests... Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtoprepo rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/altertable.sql Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql ...(.more tests).... Generated report: derbylang_report.txt |
This output does not show whether the tests passed or failed. The Summary section in <testsuite>_report.txt shows the statistics of the passed vs. failed tests, the summary <testsuite>_*.txt files list the tests that passed and failed.
4.
Harness internals for developers
The following is intended for people who have the subversion tree available and want to add or modify tests.
The test harness executing one test basically does the following in sequence:
- identify test to run
- identify properties to run with
- copy needed support files
- find the expected output
- if network server, start network server
- run the test, creating the database
- if network server, shutdown the server
- modify the output based on Sed class and _sed.properties file for the test
- compare expected output with actual output
- if pass, cleanup.
4.1 Test types
The test harness recognizes, or will recognize tests with the following
extensions:
- .java tests that run in a separate jvm.
- .sql tests that run using ij
- .sql2 related to .sql
- .multi multi threaded tests. There is
currently only 1 test being run. The multi test functions a little
differently from .java and .sql* tests in that RunTest starts a
separate harness class called MultiTest to control the details of the
run. Also, the actual test files live under
org/apache/derbyTesting/functionTests/multiTests, rather than
org/apache/derbyTesting/functionTests/tests.
- .unit unit tests. The unit tests
actually refer to <testname>_derby.properties files under
org/apache/derbyTesting/functionTests/tests/unit that activate the
actual unit test harness and tests under
org/apache/derbyTesting/unitTests. These tests test more underlying
functionality than the (rest of the) functionTests, which are more
geared toward how end-users might use functionality.
4.2 Supporting files for tests
Various additional files may be used by a test, for instance, to create
large data values, to test using of jar files and the like. Any files
that need to be accessed by a particular test that are not accessed
from the classpath need to be listed under supportfiles= in the
<testname>_app.properties file.
Tests can refer to classes without being in the classpath, and sql
tests can use the ij command 'run resource ' to execute additional .sql
files without changes to the _app.properties files.
For example, in the file
(org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:
supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar
4.3 <testname>_app.properties
Every test directory has a default_app.properties. This file is for system level properties generic to all the tests in that test directory.
If a test requires different system level properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _app.properties
For example, for the test tools/dblook_test.java, there is a
properties file called tools/dblook_test_app.properties
4.4 <testname>_derby.properties
Every test directory has a default_derby.properties. This file is for
derby specific properties common to all the tests in that test
directory.
If a test requires different derby properties, a test specific
properties file can be created to overwrite the defaults. The test
specific properties file needs to have a name starting with the
test file name, followed with _derby.properties
4.5 tmp files, out files, master files, and canons
The test's output will be put into a file testname.tmp. Then the output
is modified if masking is required and the result is put into a .out
file.
The expected output is found by examining the following directories,
based on certain input
- functionTests/master/framework/jcc_version/jvmcode
- functionTests/master/framework/jcc_version/earlier_jvmcode
- functionTests/master/framework/jcc_version
- functionTests/master/framework/jvmcode
- functionTests/master/framework/earlier_jvmcode
- functionTests/master/jvmcode
- functionTests/master
For example, if we are running a test and the flag -Dframework=DerbyNet is used and the jvm we are using is Sun's jdk 142, and the jcc version is 2.4 (not available at time of writing) then the search for the master to compare with starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with the same name as the test is found in that directory, that master is taken. If there is no such file in that directory, search continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.
If there is no file there, nor for any other jcc directory, it will
continue to derbynet/jdk14, and the search is continued for earlier jvm
versions.
If we are not running network server, the DerbyNet and
jcc_version directories are not traversed.
The version details do not go into the subversion level, i.e. running with jdk141 or jdk142 is expected to have the same behavior.
This functionality supports dealing with minor differences in behavior caused by minor differences in behavior in the underlying jvms, jcc versions, differences between results returned through network server vs. embedded and minor differences between a debug and non debug (jar) build.
However, having a large number of these files means a maintenance problem. Every time test output changes due to modifications to derby or to the test, all output files in all directories need to be updated accordingly. If at all possible, irrelevant differences should be masked out, or the test should be written so that the output does not reflect such items.
Suggestions to minimize canons:
- create test specific masking
- ensure test data has a specific correct returned order; or an order by should be added to a query
- when writing java tests, ensure only pertinent output is reflected.
4.6 Masking and comparing
Tests often fail because of unimportant differences, such as process
ids, statement ids, timestamps. The derby functional test harness
provides for masking of these differences at 2 levels:
- overall level. Masking required in all, or many tests can be achieved using the class Sed in the test harness directory. This class can either delete a reference present in the .tmp file from the .out file, or replace it with a generic string.
- test specific level. To make masking for only one test, a (testname)_sed.properties file can be created which allows to either remove a string from the output or to replace it.
The diff is executed between the final resulting output and the master
file found.
4.7 Adding a new test
To add a new test:
- create the test file (e.g. newfunctest.java or newfunctest.sql) in the appropriate tests subdirectory
- list any files needed that are not .sql or .java files in a
supportfiles entry in a test specific _app.properties file. e.g.
newfunctest_app.properties: supportfiles=xyz.jar
- list any specific derby properties in a test specific _derby.properties file.
- add the properties files to the copyfiles.ant file in the test specific directory
- run the test. The first time around, the test will fail because no master file will be found.
- if the output is correct, copy it to the master directory. Note that there is no copyfiles.ant file needed for the master directory, all .out files are automatically copied.
- run the test again. Investigate if any differences need to be masked out using a test specific sed.properties file (e.g. newfunctest_sed.properties). If so, ensure this is added to copyfiles.ant.
- add the test to a specific suites/*.xml file, maintaining proper xml syntax.
- run the suite, and correct any problems found.
4.8 Suites and Adding a new suite
A suite constitutes of a <suitename>.properties file and/or a <suitename>.runall file in the org/apache/derbyTesting/functionTests/suites directory. The .properties files hold references to other .properties files, or .runall files, the .runall files are the actual lists of tests.
The lowest level suite always needs to have a .runall file.
For example, the derbyall suite is only a derbyall.properties file that refers to other suites in the 'suites' property:
| -----------------------derbyall.properties--------------- suites=derbylang derbynetmats storeall xa derbytools derby.debug.true=enableBtreeConsistencyCheck derby.stream.error.logSeverityLevel=0 |
The derbylang suite is only a derbylang.runall, which lists the tests. The derbynetmats suite has both a .runall and a .properties file, so some additional properties can be specified that are true for all tests in that suite.
| ------------------derbynetmats.properties----------------- framework=DerbyNet suites=derbynetmats jdk12test=true runwithj9=false timeout=60 |
To add a suite, you need to create at least a <suite>.runall
file, which lists the actual tests, or a properties file that refers to
other suites that do have a .runall file. The suite should be added
into the directory
${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.
4.9
Running
with a new jvm
Currently, the supported jvms are:
| jdk131
- Sun
HotSpot jdk1.3.1 - class: jdk13 jdk141 - Sun HotSpot jdk1.4.1 - class jdk14 jdk142 - Sun HotSpot jdk1.4.2 - class jdk14 jdk15 - Sun HotSpot jdk1.5 - class jdk15 ibm131 - IBM Classic jdk1.3.1 - class ibm13 ibm141 - IBM Classic jdk1.4.1 - class ibm14 ibm142 - IBM Classic jdk1.4.1 - class ibm14 j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition) - class j9_13 |
The classes above are subclasses of
org.apache.derbyTesting.functionTests.harness.jvm. The name at the
front is just a convention.
To run a test with a jvm that does not have a matching class under
org.apache.derbyTesting.functionTests.harness, do the following:
- just run the tests as if there is a jvm class. The harness will
default to using
the jdk14 class. Unlikely, but possibly there are no differences
- if there are failures showing that cannot be explained any other way but genuine, acceptable jvm differences, do the following:
- create a subclass of org.apache.derbyTesting.functionTests.harness.jvm. In this class, specify any jvm specific property settings required
- compile the new jvm class and run the tests
- create a new canon directory for any additional canons that need to be created.
- in rare occasions, other harness changes may be required
- for any tests that should not run with this environment, add a line in the testname_app.properties file indicating this. For instance to add a line for a jvm called jdk29, it would be like this: runwithjdk29=false. Note that the versioning does not currently extend past 2 digits.
- Add code in RunTest.java to switch to the new jvm based on values for system and vendor properties
4.10 Skipping a test
There are 2 skipping mechanisms in place for different kinds of
skipping of a test.
Some tests are written to test specific functionality only available
with for instance certain jvms, or, with network server, certain
versions of the IBM Universal Driver. To control this, properties can
be set for each test, for instance, if a test should not be run when
using an ibm jvm, set runwithibmjvm=false. If a test should be run with
Sun Hotspot jvm version 14, then set runwithjdk14=true.
The skip setting does not go into the subversion level, i.e. setting
runwithjdk141=false has no effect, and setting runwithjdk14 affects
runs with jdk141 as well as jdk142.
Other skip reasons are encryption protocols specific to a certain
jvm.
The property for skipping a test based on the version of the IBM
Universal Driver is "excludeJCC". The keywords "at-or-before" and "at-or-after" can be used to specify
which range of JCC versions should be excluded. If neither of
these keywords is provided, the default is "at-or-before". For example:
excludeJCC=at-or-before:2.4
To skip a test when running with any version of the IBM Universal Driver that is 2.0 or later:
excludeJCC=at-or-after:2.0
You can also specify an (optional) jvm clause to further tune the
exclusion criteria. This clause starts with the ",when" tag and is followed by a
three-part jvm version. In this case, a test will only be skipped
if BOTH the JCC clause AND the jvm clause are true. For example:
To skip a test when running with any version of the IBM Universal
Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1
excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1
Another skipping mechanism works on entire 'frameworks'. Currently there are only 3 supported in the harness, embedded, NetworkServer with the IBM Universal JDBC Driver ('DerbyNet') and NetworkServer with the not yet available new driver ('DerbyNetClient'). In the suites directory there are .exclude files for each of the frameworks - if for some reason an exclude file were not there, you would see a warning message show up for every test run. In this 'framework'.exclude file tests can be placed that for some reason need to be excluded from running with that framework. This mechanism enables adding of suites to a framework run even if a few of the tests are not appropriate for that particular framework.
Note that at this time, only skipped suites show up in the .skip result file. This still needs to be corrected.
4.11 Frameworks
Currently, the only framework used is DerbyNet for network server.
Setting framework property setting will invoke the test harness class
NetServer, which currently has configuration for connecting to DB2 via
jcc (the IBM Universal Driver), and via the older db2 driver. But there
are currently no tests to exercise these settings.
Setting this framework also causes the search for expected output to
include DerbyNet and jcc version specific subdirectories under master.
4.12 Some test harness properties
For a complete set, refer to comments in RunTest.java, but here are some valuable test properties which can be passed to the RunTest class:
| runwith<jvm> See above section 4.10 framework specifies which framework to run with. For example: java -Dframework=DerbyNet org.apache.derbyTesting.functionTests.RunTest lang/supersimple.sql verbose Shows more detailed output. For example: java -Dverbose=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql keepfiles Indicates to not clean up any of the files if the test passed. java -Dkeepfiles=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql TestSpecialFlags sets additional properties. java -D=TestSpecialFlags=derby.infolog.append=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql excludeJCC See above section 4.10 |
Notes
Note1:The j9derbynetmats suite is included to serve as an example of splitting the network server process to run with a different jvm than the test client. The j9derbynetmats suite will run with another jvm as client (as defined in the suite properties), but start up network server with the 'j9' jvm files (the reference to 'j9' is based on the executable, j9.exe), based on the property 'serverJvm'. Running this suite requires providing the property bootcp, which is interpreted from the test harness class j9_13. See also section on adding a new jvm setup.
Another suite is derbynetclientmats. This suite is for testing the networkserver with a yet to be contributed open source client. IBM staff is working on contributing this. In essence it is the same as the derbynetmats suite. At this time, it can only be run by the IBM staff working on this driver.
Note2:
Note3:
Warning: Cleanup failed on baseDir: /local/myrun1/DerbyNet/supersimple
