Hello,
I know that it's been common practice for developers on Xalan-Java to
run the test build.bat or build.sh script, using the target smoketest or
smoketest.xsltc, as appropriate. Those build targets use the "trax"
flavor, which is equivalent to "trax.systemId", I believe.
However, there are sometimes subtle differences between the results
produced using any two of the (31-derful) flavors available for running
tests. On a number of occasions, regressions have slipped in that could
only be detected using one flavor. I'd like to suggest that developers
use the "alltest" target for Xalan-J Interpretive and "alltest.conf.xsltc"
for XSLTC (and possibly "alltest.accept.xsltc"), as appropriate. These
will run conformance tests using all six flavors, and in the case of
Xalan-J Interpretive, will run other tests as well.
Some time ago, Gordon Chiu created a stylesheet that can be used to
compare results between two test runs, but I don't believe it was
well-advertised. If you have run the "alltest.conf.xsltc" target before
and after making a change, for instance, and the two sets of results are
in the directories "results-alltest.xsltc" and
"results-alltest.xsltc.old", you can generate a comparison using the
following commands (translate as appropriate for your operating
environment):
set RESULTSCANNER=tableResults.xsl
viewResults.bat results-alltest.xsltc/conf/sax/results.xml \
comparison.html \
-param resultDir results-alltest.xsltc/conf \
-param compareAgainst results-alltest.xsltc.old/conf
The results will be placed in comparison.html.
I've attached an example of the tabular results that are generated.
The columns contain the results for each flavor, the rows contain the
results for each subdirectory. A legend at the bottom of the table
describes how to read the "fail/err" and "pass" rows in the table. Red
text and magenta text is neutral, but bold red is bad! Green text that is
struck through is good.
If the use of colour and bold type presents an accessibility problem
for anybody, we can look at other other ways of highlighting regressions
and new successes.
I know that there are many more things that you can do with the
viewResults tool and appropriate stylesheets. This is one use that I've
found particularly beneficial.
Thanks,
Henry
------------------------------------------------------------------
Henry Zongaro Xalan development
IBM SWS Toronto Lab T/L 969-6044; Phone +1 905 413-6044
mailto:[EMAIL PROTECTED]
| Conformance Test Results: conf | |||||||
| Environment |
Currently Tested:
Xalan Java 2.5.2 (Tue Feb 24 09:19:41 EST 2004) and Xerces-J 2.6.2 IBM Corporation JDK 1.3.1 Test run by: henry on a x86 running Windows 2000 Processor: XalanJ-I Compared Against: Xalan Java 2.5.2 (Mon Feb 23 22:32:30 EST 2004) and Xerces-J 2.6.2 IBM Corporation JDK 1.3.1 Test run by: henry on a x86 running Windows 2000 Processor: XalanJ-I |
||||||
| Overall | Flavour: | trax.dom | trax.sax | trax.stream | trax.file | trax.systemId | trax.localPath |
| Categories | |||||||
| Total: | 35 | 35 | 35 | 35 | 35 | 35 | |
| Pass: | 19 | 29 | 30 | 30 | 30 | 30 | |
| Fail: | 4 | 4 | 4 | 4 | 4 | 4 | |
| Error: | 12 | 2 | 1 | 1 | 1 | 1 | |
| Test Cases | |||||||
| Total: | 1688 | 1689 | 1689 | 1689 | 1689 | 1689 | |
| Pass: | 1630 | 1677 | 1680 | 1680 | 1680 | 1680 | |
| Fail: | 11 | 8 | 8 | 8 | 8 | 8 | |
| Error: | 47 | 4 | 1 | 1 | 1 | 1 | |
| Category Summaries | |||||||
| attribset | |||||||
| #cases | 49 | 49 | 49 | 49 | 49 | 49 | |
| pass | 01-03, 05-08, 10-16, 18-20, 22-39, 41-49 | 01-39, 41-49 | 01-39, 41-49 | 01-39, 41-49 | 01-39, 41-49 | 01-39, 41-49 | |
| fail/err | 04 09 17 21 40 | 40 | 40 | 40 | 40 | 40 | |
| attribvaltemplate | |||||||
| #cases | 12 | 12 | 12 | 12 | 12 | 12 | |
| pass | 01-06, 09-13 | 01-06, 08-13 | 01-06, 08-13 | 01-06, 08-13 | 01-06, 08-13 | 01-06, 08-13 | |
| fail/err | 08 | ||||||
| axes | |||||||
| #cases | 130 | 130 | 130 | 130 | 130 | 130 | |
| pass | 01-61, 63-115, 117, 119-131 | 01-61, 63-117, 119-131 | 01-61, 63-117, 119-131 | 01-61, 63-117, 119-131 | 01-61, 63-117, 119-131 | 01-61, 63-117, 119-131 | |
| fail/err | 62 116 | 62 | 62 | 62 | 62 | 62 | |
| boolean | |||||||
| #cases | 90 | 90 | 90 | 90 | 90 | 90 | |
| pass | 01-90 | 01-90 | 01-90 | 01-90 | 01-90 | 01-90 | |
| conditional | |||||||
| #cases | 23 | 23 | 23 | 23 | 23 | 23 | |
| pass | 01-23 | 01-23 | 01-23 | 01-23 | 01-23 | 01-23 | |
| conflictres | |||||||
| #cases | 36 | 37 | 37 | 37 | 37 | 37 | |
| pass | 01-18, 20-37 | 01-37 | 01-37 | 01-37 | 01-37 | 01-37 | |
| fail/err | (19) | ||||||
| copy | |||||||
| #cases | 62 | 62 | 62 | 62 | 62 | 62 | |
| pass | 01-20, 22-55, 57, 61-62 | 01-55, 57, 61-62 | 01-55, 57, 61-62 | 01-55, 57, 61-62 | 01-55, 57, 61-62 | 01-55, 57, 61-62 | |
| fail/err | 21 56 58 59 60 | 56 58 59 60 | 56 58 59 60 | 56 58 59 60 | 56 58 59 60 | 56 58 59 60 | |
| dflt | |||||||
| #cases | 4 | 4 | 4 | 4 | 4 | 4 | |
| pass | 01-04 | 01-04 | 01-04 | 01-04 | 01-04 | 01-04 | |
| embed | |||||||
| #cases | 5 | 5 | 5 | 5 | 5 | 5 | |
| pass | 01-04, 07 | 03-04 | 01-04, 07 | 01-04, 07 | 01-04, 07 | 01-04, 07 | |
| fail/err | 01 02 07 | ||||||
| _expression_ | |||||||
| #cases | 6 | 6 | 6 | 6 | 6 | 6 | |
| pass | 01-06 | 01-06 | 01-06 | 01-06 | 01-06 | 01-06 | |
| extend | |||||||
| #cases | 5 | 5 | 5 | 5 | 5 | 5 | |
| pass | 01-05 | 01-05 | 01-05 | 01-05 | 01-05 | 01-05 | |
| idkey | |||||||
| #cases | 62 | 62 | 62 | 62 | 62 | 62 | |
| pass | 01-13, 15-30, 32-63 | 01-13, 15-30, 32-63 | 01-13, 15-30, 32-63 | 01-13, 15-30, 32-63 | 01-13, 15-30, 32-63 | 01-13, 15-30, 32-63 | |
| fail/err | 31 | 31 | 31 | 31 | 31 | 31 | |
| impincl | |||||||
| #cases | 29 | 29 | 29 | 29 | 29 | 29 | |
| pass | 01-29 | 01-29 | 01-29 | 01-29 | 01-29 | 01-29 | |
| lre | |||||||
| #cases | 23 | 23 | 23 | 23 | 23 | 23 | |
| pass | 01-12, 14-23 | 01-23 | 01-23 | 01-23 | 01-23 | 01-23 | |
| fail/err | 13 | ||||||
| match | |||||||
| #cases | 32 | 32 | 32 | 32 | 32 | 32 | |
| pass | 01-32 | 01-32 | 01-32 | 01-32 | 01-32 | 01-32 | |
| math | |||||||
| #cases | 107 | 107 | 107 | 107 | 107 | 107 | |
| pass | 01-105, 110 | 01-105, 110-111 | 01-105, 110-111 | 01-105, 110-111 | 01-105, 110-111 | 01-105, 110-111 | |
| fail/err | 111 | ||||||
| mdocs | |||||||
| #cases | 19 | 19 | 19 | 19 | 19 | 19 | |
| pass | 05-07, 09-10, 12-14, 16-18 | 01-19 | 01-19 | 01-19 | 01-19 | 01-19 | |
| fail/err | 01 02 03 04 08 11 15 19 | ||||||
| message | |||||||
| #cases | 16 | 16 | 16 | 16 | 16 | 16 | |
| pass | 01-16 | 01-16 | 01-16 | 01-16 | 01-16 | 01-16 | |
| modes | |||||||
| #cases | 17 | 17 | 17 | 17 | 17 | 17 | |
| pass | 01-17 | 01-17 | 01-17 | 01-17 | 01-17 | 01-17 | |
| namedtemplate | |||||||
| #cases | 19 | 19 | 19 | 19 | 19 | 19 | |
| pass | 01-19 | 01-19 | 01-19 | 01-19 | 01-19 | 01-19 | |
| namespace | |||||||
| #cases | 134 | 134 | 134 | 134 | 134 | 134 | |
| pass | 01-02, 04-48, 52-75, 77-84, 86-87, 89-100, 102-106, 110-113, 116-117, 119-128, 130-132, 134, 137-142 | 01-75, 77-84, 86-113, 115-117, 119-128, 130-132, 134, 137-142 | 01-75, 77-84, 86-113, 115-117, 119-128, 130-132, 134, 137-142 | 01-75, 77-84, 86-113, 115-117, 119-128, 130-132, 134, 137-142 | 01-75, 77-84, 86-113, 115-117, 119-128, 130-132, 134, 137-142 | 01-75, 77-84, 86-113, 115-117, 119-128, 130-132, 134, 137-142 | |
| fail/err | 03 49 50 51 88 101 107 108 109 115 | ||||||
| node | |||||||
| #cases | 21 | 21 | 21 | 21 | 21 | 21 | |
| pass | 01-16, 18-21 | 01-21 | 01-21 | 01-21 | 01-21 | 01-21 | |
| fail/err | 17 | ||||||
| numberformat | |||||||
| #cases | 44 | 44 | 44 | 44 | 44 | 44 | |
| pass | 01-09, 11-32, 34-46 | 01-09, 11-32, 34-46 | 01-09, 11-32, 34-46 | 01-09, 11-32, 34-46 | 01-09, 11-32, 34-46 | 01-09, 11-32, 34-46 | |
| numbering | |||||||
| #cases | 93 | 93 | 93 | 93 | 93 | 93 | |
| pass | 01-14, 17-95 | 01-14, 17-95 | 01-14, 17-95 | 01-14, 17-95 | 01-14, 17-95 | 01-14, 17-95 | |
| output | |||||||
| #cases | 109 | 109 | 109 | 109 | 109 | 109 | |
| pass | 01-25, 27-30, 32-71, 73-75, 77, 80-82, 84-85, 87-107, 114 | 01-30, 32-77, 80-82, 84-107, 109-114 | 01-30, 32-77, 80-82, 84-107, 109-114 | 01-30, 32-77, 80-82, 84-107, 109-114 | 01-30, 32-77, 80-82, 84-107, 109-114 | 01-30, 32-77, 80-82, 84-107, 109-114 | |
| fail/err | 26 72 76 86 109 110 111 112 113 | ||||||
| position | |||||||
| #cases | 113 | 113 | 113 | 113 | 113 | 113 | |
| pass | 01-110 | 01-111 | 01-111 | 01-111 | 01-111 | 01-111 | |
| fail/err | 111 112 113 | 112 113 | 112 113 | 112 113 | 112 113 | 112 113 | |
| predicate | |||||||
| #cases | 58 | 58 | 58 | 58 | 58 | 58 | |
| pass | 01-58 | 01-58 | 01-58 | 01-58 | 01-58 | 01-58 | |
| processorinfo | |||||||
| #cases | 1 | 1 | 1 | 1 | 1 | 1 | |
| pass | 03 | 03 | 03 | 03 | 03 | 03 | |
| reluri | |||||||
| #cases | 11 | 11 | 11 | 11 | 11 | 11 | |
| pass | 01-08, 10-11 | 01-11 | 01-11 | 01-11 | 01-11 | 01-11 | |
| fail/err | 09 | ||||||
| select | |||||||
| #cases | 85 | 85 | 85 | 85 | 85 | 85 | |
| pass | 01-06, 08-45, 47-86 | 01-45, 47-86 | 01-45, 47-86 | 01-45, 47-86 | 01-45, 47-86 | 01-45, 47-86 | |
| fail/err | 07 | ||||||
| sort | |||||||
| #cases | 38 | 38 | 38 | 38 | 38 | 38 | |
| pass | 01-16, 20-21, 26-29, 32-34, 36, 38-41 | 01-16, 20-41 | 01-16, 20-41 | 01-16, 20-41 | 01-16, 20-41 | 01-16, 20-41 | |
| fail/err | 22 23 24 25 30 31 35 37 | ||||||
| string | |||||||
| #cases | 135 | 135 | 135 | 135 | 135 | 135 | |
| pass | 01-02, 04-22, 30-143 | 01-02, 04-22, 30-143 | 01-02, 04-22, 30-143 | 01-02, 04-22, 30-143 | 01-02, 04-22, 30-143 | 01-02, 04-22, 30-143 | |
| variable | |||||||
| #cases | 70 | 70 | 70 | 70 | 70 | 70 | |
| pass | 01-50, 52-70 | 01-70 | 01-70 | 01-70 | 01-70 | 01-70 | |
| fail/err | 51 | ||||||
| ver | |||||||
| #cases | 8 | 8 | 8 | 8 | 8 | 8 | |
| pass | 01-08 | 01-08 | 01-08 | 01-08 | 01-08 | 01-08 | |
| whitespace | |||||||
| #cases | 22 | 22 | 22 | 22 | 22 | 22 | |
| pass | 01-13, 15, 17-23, 35 | 01-13, 15, 17-23, 35 | 01-13, 15, 17-23, 35 | 01-13, 15, 17-23, 35 | 01-13, 15, 17-23, 35 | 01-13, 15, 17-23, 35 | |
| Legend | |||||||
|
|||||||
