I think that we should use the XQTS test suite as much as possible.
It is a great opportunity for us that we already have a test suite with
a good coverage of the language features.
As we currently do not pass all tests, I think that we could create a
file that indicates which tests pass and use this to run regression
tests for each build. One open question is, if the file should contain
the known failures or the know successes. In any case we need to
manually modify that file (or at least re-generate it), whenever we
improve out test coverage.
I think that the fact that we need to download the XQTS is not too
worrying. We can either try to automate that (which has been problematic
in the past) or we can keep the download manual and skip the tests, if
the XQTS is not available. That way anybody could download, build and
run VXQuery without XQTS, but we should obviously run it before
submitting the the repository. (And ideally we would automate the
execution of the XQTS tests for any commit that goes into the master
branch).
Does this approach make sense?
Cheers,
Till
On 5 Feb 2015, at 16:31, Eldon Carman wrote:
How should we approach adding tests to mvn to ensure our build is not
breaking XQuery functionality? The tests must pass to be added to the
build
process. The current build tests include parallel tests that are not
covered in the XML Query Test Suite (XQTS). The XQTS has 20,000 tests
that
cover XQuery specification although currently only about half of the
tests
pass in VXQuery. In addition, running all those tests takes a few
minutes.
How can we ensure quality maven builds during development with the
inclusion of more XQuery testing? Off the top of my head, we have two
options:
1. We could pick and choose small test groups from of the XQTS test
suite
to run.
+ Test have already been written.
- XQTS must be downloaded.
- Tests may not line up with build test goals.
2. Write our own small test list for basic XQuery functionality.
+ Tests will be created specifically for VXQuery.
- Must create all tests.
In both cases, the full XQTS test suite would be used for a more
thorough
and complete testing.
What do you think?
