+1 Also, you can take a look at Zorba <http://zorba.io>and inspire from its scripts <http://bazaar.launchpad.net/~zorba-coders/zorba/trunk/files/head:/test/rbkt/Scripts/w3c/> on running automated XQTS.
Cezar On Fri, Feb 6, 2015 at 7:50 PM, Till Westmann <[email protected]> wrote: > I think that we should use the XQTS test suite as much as possible. > It is a great opportunity for us that we already have a test suite with a > good coverage of the language features. > > As we currently do not pass all tests, I think that we could create a file > that indicates which tests pass and use this to run regression tests for > each build. One open question is, if the file should contain the known > failures or the know successes. In any case we need to manually modify that > file (or at least re-generate it), whenever we improve out test coverage. > > I think that the fact that we need to download the XQTS is not too > worrying. We can either try to automate that (which has been problematic in > the past) or we can keep the download manual and skip the tests, if the > XQTS is not available. That way anybody could download, build and run > VXQuery without XQTS, but we should obviously run it before submitting the > the repository. (And ideally we would automate the execution of the XQTS > tests for any commit that goes into the master branch). > > Does this approach make sense? > > Cheers, > Till > > > On 5 Feb 2015, at 16:31, Eldon Carman wrote: > > How should we approach adding tests to mvn to ensure our build is not >> breaking XQuery functionality? The tests must pass to be added to the >> build >> process. The current build tests include parallel tests that are not >> covered in the XML Query Test Suite (XQTS). The XQTS has 20,000 tests that >> cover XQuery specification although currently only about half of the tests >> pass in VXQuery. In addition, running all those tests takes a few minutes. >> >> How can we ensure quality maven builds during development with the >> inclusion of more XQuery testing? Off the top of my head, we have two >> options: >> 1. We could pick and choose small test groups from of the XQTS test suite >> to run. >> + Test have already been written. >> - XQTS must be downloaded. >> - Tests may not line up with build test goals. >> 2. Write our own small test list for basic XQuery functionality. >> + Tests will be created specifically for VXQuery. >> - Must create all tests. >> >> In both cases, the full XQTS test suite would be used for a more thorough >> and complete testing. >> >> What do you think? >> >
