Hi Dave,
Those are good points.
Products I've worked on in the past have had many, many extensions
added over many releases. The set of tests that worked in a particular
way today and should have worked in the same way on all previous versions
of the product was exceedingly small, so we always had complete snapshots
of tests that corresponded to particular versions of the product. But
when the bulk of tests that work today should have worked on all preceding
versions of the processor, I agree that the tagging that I thought would
be necessary for (1) shouldn't be - at least for the conformance cluster.
However, within the acceptance cluster, aren't we likely to run into
a situation in which the way a processor handles a particular feature
changes from one release to another? In such a situation wouldn't we need
tagged versions of the acceptance cluster gold files that would be
associated with particular versions of Xalan or XSLTC? If not, then I
don't think I've understood the nature of the acceptance cluster.
Thanks,
Henry
------------------------------------------------------------------
Henry Zongaro Xalan development
IBM SWS Toronto Lab Tie Line 969-6044; Phone (905) 413-6044
mailto:[EMAIL PROTECTED]
David Marston/Cambridge/IBM@Lotus
11/28/2002 12:19 PM
Please respond to xalan-dev
To: [EMAIL PROTECTED]
cc:
Subject: Re: conf test restructuring
Henry Zongaro writes:
>If we go with separate tags for the accept-gold tree, we'll end up with
>multi-level tagging - at least conceptually. When we put out a Xalan-J
>2.7.9 release, say, we'll have a xalanj279 tag for the source and some
>of the tests, and tags like xalanj279-accept-gold-xalan-interpretive
>xalanj279-accept-gold-xsltc for the different accept silver trees.
That seems wrong from a Quality Engineering perspective. Taking it from
general principles down to specifics:
The "conf" (conformance) cluster should be a set of tests for which all
XSLT processors produce the same results, after accounting for the
discretionary items. The "accept" (acceptance) cluster is a set of tests
which Xalan must pass in order to be deemed good enough to ship. In both
areas, the tests themselves represent what Xalan should have done ever
since Day One and should continue to do forever, thus there is no need
to apply a version number to these tests. As you've noticed, I've added
new tests in the past couple weeks, some of which Xalan has been passing
for a long time and some of which reveal bugs that have existed for a
long time. Still, the tests themselves are timeless.
I propose to accommodate the XSLT/XPath 2.0 standards in parallel
"conf2" and "accept2" clusters. A very few behaviors will change in a
non-compatible way, but the XSL Working Group of the W3C seems to be
heading toward a "1.0 compatibility module" under which processors
will react to version="1.0" in the stylesheet by behaving exactly as
an XSLT 1.0 processor. Since Xalan would implement this module, that
means the XSLT 2.0 version will need to pass all of conf2, accept2,
conf, and accept, the latter two via the compatibility module.
We account for known bugs through the test.properties file, and that
probably needs to be versioned for the various processors and
whatever else is spawning a separate accept-silver tree. But the tests
themselves can be applied to all flavors, even if the silver files
have to be different. (Note that this statement applies to accept but
not the extension tests.) A person developing a patch needs to check
out conf and accept tests, start with code that passes all such tests
as filtered by test.properties, make their patch, and still pass all
the (filtered) conf and accept tests just as cleanly after the patch.
If the patch is a fix for a known bug, then presumably the patched
version passes at least one more conf or accept test, which can then
be de-listed from the exclude list in test.properties. Subsequent
patches will need to pass a slightly larger test battery. (I know
that most readers of this list already know this policy, but I can't
recall when it was last written out like this.)
.................David Marston
