I agree, the test framework is far from obvious. Tests for open issues do indeed go into the jira directory, as you have found, and are executed with "build jira". When the fix is confirmed, they get moved into the other directories. (The "bugzilla" directory is a similar temporary stash, left over from when we were using bugzilla rather than jira.)
Unfortunately, for historical reasons, the structure of the other directories is a bit different. 1. They're divided by general category of test: Acceptance, API, conf(ormance), contrib(uted by users), exslt (build-in extended features), extensions (the user extension facilities), per(ormance) (rarely run), and tests expected to fail or to "correctly" throw errors. These categories may in turn have subcategories, represented by subdirectories. 2. The input files go in X, and the expected output files go in X-gold. For example, a testcase for whitespace puts its input .xml and .xsl files in /accept/spacing, and the expected output files are in accept-gold/spacing. The drivers automagically reference the appropriate directories to match these up. The categories of tests are grouped together into test suites in the .ant build file. There is a mechanism for "we know this test fails because we have chosen to diverge from the standard for now"... But I honestly don't remember the details offhand. The XSLTC tests really want to be updated to use that properly. Hope that helps. Legacy code. "The impressive thing is not how well the bear dances, but that it dances at all." -- ` /_ Joe Kesselman (he/him/his) -/ _) My Alexa skill for New Music/New Sounds fans: / https://www.amazon.com/dp/B09WJ3H657/ Caveat: Opinionated old geezer with overcompensated writer's block. May be redundant, verbose, prolix, sesquipedalian, didactic, officious, or redundant. Feel free to call him on it. ________________________________ From: Andreas Martens <amart...@uk.ibm.com> Sent: Wednesday, April 2, 2025 11:48 AM To: dev@xalan.apache.org <dev@xalan.apache.org> Subject: Re: three PRs and counting OK, I’ve stared at xalan-test for a while now and I still don’t understand it enough to write a new test. I can run smoketest, apitest and conf.xsltc and see that I haven’t increased (or decreased) the failures of existing tests. But I can’t see where and how to add a new test, the documentation (https://xml.apache.org/xalan-j/test/overview.html) doesn’t seem to match what’s in the repository, the new test section says to open org.apache.qetest.trax.REPLACE_template_for_new_tests.java which doesn’t seem to exist… I was hoping there would be somewhere I could drop in my sets of xsl+xml+output and have it run both xsltc and the interpreted xalan over them. The closest place I could see was under tests/contrib, but I don’t think I can merely add to them. If I started a new folder structure like: /tests/regression/xalanj-2785/example1.xml example1.xsl regression-gold/xalanj-2785/example1.out would that be appropriate? I see there’s a jira subdir, is that what that’s for? It’s kinda empty… And I can’t seem to run it, it’s referenced by alltest.other, but that doesn’t run because thre’s no api target in testxsl. Ah-hah, if I name files “jira_” + something in the jira folder then they’ll get run when running `./build.sh jira`! But if that doesn’t actually get run by anything else, is it of value? I was trying to find something to copy-paste really, but it’s hard to tell how things are used, e.g. there’s a directory /tests/contrib/garypeskin/ with a bunch of xml+xsl+out that looked like something I could start with, but this isn’t referenced anywhere, so has it been run? I created a bunch of test in the jira dir, and was really happy when they passed, I then removed my fixes from the xalan-java repo and rebuilt, and to my horror the tests still passed! It took me a while to realise they’re using the Interpreter and not the Compiler! So, to get further, I hacked build.xml to add <!-- Explicitly set TransformerFactory property to use xsltc --> <param name="sysprop.transfact" value="org.apache.xalan.xsltc.trax.TransformerFactoryImpl" /> which got my new tests running with xsltc, now failing as expected. And when I flip to using my new changes, I can get them to pass (though the output is different to what the Interpreter outputs in a couple notable cases). So, to conclude: * Where and how do I add my tests? Thanks, Andreas From: Andreas Martens <amart...@uk.ibm.com> Date: Tuesday, 1 April 2025 at 15:50 To: dev@xalan.apache.org <dev@xalan.apache.org> Subject: [EXTERNAL] Re: three PRs and counting Thanks both Greg and Joe, I’ve got xalan-test and all the important tests pass, I’ll get up to speed on writing new tests next! Since it’s in a separate repo I’ll create a separate PR for them when they’re ready :-) Cheers, Andreas From: Joseph Thanks both Greg and Joe, I’ve got xalan-test and all the important tests pass, I’ll get up to speed on writing new tests next! Since it’s in a separate repo I’ll create a separate PR for them when they’re ready :-) Cheers, Andreas From: Joseph Kesselman <kesh...@alum.mit.edu.INVALID> Date: Monday, 31 March 2025 at 14:20 To: dev@xalan.apache.org <dev@xalan.apache.org> Subject: [EXTERNAL] Re: three PRs and counting The test suite for Xalan still lives in a separate Apache project, xalan-test, since it had ambitions of being a test suite shared by other implementations of XSLT. If you downloaded one of the archive files (jar or zip), a copy of xalan-test The test suite for Xalan still lives in a separate Apache project, xalan-test, since it had ambitions of being a test suite shared by other implementations of XSLT. If you downloaded one of the archive files (jar or zip), a copy of xalan-test as of that release date should have been included; you can always fetch the current version from git. The test framework is still based on Ant rather than Maven. Note that some of the tests are known and accepted failures, where Xalan is known not to conform precisely to the formal XSLT Recommendation. The drivers should both report failure on those individual tests and success on the overall test run. Yes, any behavioral changes checked into the Xalan code base should be accompanied by new tests in xalan-tests to demonstrate the difference between old and new code and that the new function is working as expected, and of course should be shown not to break existing tests. -- /_ Joe Kesselman (he/him/his) -/ _) My Alexa skill for New Music/New Sounds fans: / https://www.amazon.com/dp/B09WJ3H657/ Caveat: Opinionated old geezer with overcompensated writer's block. May be redundant, verbose, prolix, sesquipedalian, didactic, officious, or redundant. ________________________________ From: Gary Gregory <garydgreg...@gmail.com> Sent: Monday, March 31, 2025 5:46:16 AM To: dev@xalan.apache.org <dev@xalan.apache.org> Subject: Re: three PRs and counting Hello Andreas, Thank you for your report and work. I am traveling for the next couple of days so I can't help much. You're in the right place to ask for help. Yes, adding tests to cover fixes and changes is important, otherwise reverting behavior is possible without realizing it. Gary On Mon, Mar 31, 2025, 10:02 Andreas Martens <amart...@uk.ibm.com<mailto:amart...@uk.ibm.com>> wrote: Hello all! Just been going through some of the problems our customers have hit whilst using our XSLT functionality that’s now underpin by XSLTC rather than IBMs XL-TXE. * https://github.com/apache/xalan-java/pull/189 fixes an unlimited growth of namespaces used within XSLTC, not a problem for short-lived programs, but we tend to run for months and years continuously processing! * https://github.com/apache/xalan-java/pull/192 fixes what I suspect was an accidental removal in XALANJ-2108 (in 2006!) of some code that ensured the variable vector was closed, which was discovered shortly after in 2008 in XALANJ-2430 (and more recently by our customer…) * https://github.com/apache/xalan-java/pull/193 adds some functionality to one of the shipped samples that I’ve found useful in debugging the two above, on the vein of if I found it useful, perhaps someone else did too! I ignored the one where the customer was referencing `msxsl:node-set($var)/Namespace` from `xmlns:msxsl="urn:schemas-microsoft-com:xslt"`, claiming that “it used to work” I presume there are some regression tests in there somewhere, so I’m going to see if I can find them and add my new fixes to them, so should hopefully come with another PR for that soon (or even better, extend existing PRs if I get there before reviews).. Anyway, how can I get these PRs reviewed and either approved or rejected? The last two are new, but the first one is from September, so I’d like to know what to do with it before I completely forget what I did :-) Cheers, Andreas -- Andreas Martens [signature_558150371] Senior Engineer App Connect Enterprise IBM Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
