Re: Running tests with CMake
On 27. 5. 2026 10:09, Timofei Zhakov wrote:
On Tue, May 26, 2026 at 9:27 PM Branko Čibej wrote:
On 26. 5. 26 20:05, Branko Čibej wrote:
I was looking at the test output of the CMake tests recently, and
I noticed that they don't run XML validation even when it's not
explicitly disabled. Looking at CMakeLists.txt, I found that each
test program is run directly instead of through run_tests.py.
I don't think that's a very good idea. Sure, developers can run
individual tests like that, but run-tests.py adds some common
semantics that the build system shouldn't just skip. Like, for
example, running XML validation on command output. It also
handles test parallelization. I understand that the CMake project
can run each test function from all the tests separately, but
it's possible to do that through run-tests.py.
The alternative would probably be to install python-lxml and
python-rnc2rng on the Windows test runners. I already did that
for Linux runners (both autotools and cmake), because it is a bit
more controlled than creating a virtual environment on the fly.
However, I don't (want to?) know now to do that with vcpkg.
Never mind. It was enough to run 'python -m pip' to install.
Tests with cmake are basically unfinished...
The problem is ctest (the utility that cmake advises to use for
testing and what add_test command generates data for) is pretty much
unusable. Sure, you can add tests and execute them, it even supports
parallel execution and running specific tests, but there are
limitations. There is no easy way to pass arguments to the test
programs and no setup action.
Now as I rethink that, it's probably a good idea to add a script
similar to win-tests.py that runs tests when built with cmake. It
would be also nice to add itself to either generator target or to ctest.
There is an issue that if we want it to be possible to run the tests
via this script directly (from the root of the source tree or
something like that) it needs to be aware of where cmake was
configured which with out-of-tree build is any directory on the
filesystem. This script could accept this directory as an argument
which also defaults to something like 'out' or 'out/build' and could
be potentially cached for convenience.
You're aware, of course, that both CMake and autotools already support
out-of-tree builds, right? In fact, that's the only way I use them.
It might also be a good idea to factor out the part that starts an
http server from win-tests.py into a separate python file to reuse it
here and maybe even merge with davautocheck.sh.
Making the {svnserve,dav}autocheck logic more portable would be nice.
Rewriting those scripts in Python would not be enough, though. I haven't
tested on Windows in literally decades and have completely forgotten how
that works.
I also wonder why we don't use python's native unittest framework [1]?
Now it's probably too late to suggest that but rather as a crazy idea,
how about we switch to it? My reasoning would be so we don't reinvent
the wheel. The wheel that worked well enough for a long time, but
still. By using a standard framework like this some IDEs might
automatically discover tests and display them in the UIs. I think it's
nice to have every test as a first class entity if it makes sense that
"adapters" can load, run, and list in whatever way they want. I'm
pretty sure that swig tests use this framework instead of svntest.
Ahahaha. Yes. We don't use unittest because it didn't exist back in the
day. There were some external testing frameworks for Python but none of
them came close to what we needed. I think the only marginally usable
one was nose. So we wrote our own, and I will never get over regretting
my complicity in that heinous act.
Thoughts?
Rewriting the test for unittest would of course be a H-H-H-UGE task. It
would also give us a lot of other benefits and flexibility; for example,
support for parallel testing is trivial, integration with test coverage
tools is trivial, and so on. But it's not some summer project.
Making a better test driver for Windows ... actually: why couldn't the
CMake build use win-tests.py? At least until some better alternative
comes along? You'd lose the ability to run individual tests, that's true.
-- Brane
[1] https://docs.python.org/3/library/unittest.html
--
Timofei Zhakov
Re: Running tests with CMake
On Tue, May 26, 2026 at 9:27 PM Branko Čibej wrote: > On 26. 5. 26 20:05, Branko Čibej wrote: > > I was looking at the test output of the CMake tests recently, and I > noticed that they don't run XML validation even when it's not explicitly > disabled. Looking at CMakeLists.txt, I found that each test program is run > directly instead of through run_tests.py. > > I don't think that's a very good idea. Sure, developers can run individual > tests like that, but run-tests.py adds some common semantics that the build > system shouldn't just skip. Like, for example, running XML validation on > command output. It also handles test parallelization. I understand that the > CMake project can run each test function from all the tests separately, but > it's possible to do that through run-tests.py. > > The alternative would probably be to install python-lxml and > python-rnc2rng on the Windows test runners. I already did that for Linux > runners (both autotools and cmake), because it is a bit more controlled > than creating a virtual environment on the fly. However, I don't (want to?) > know now to do that with vcpkg. > > > > Never mind. It was enough to run 'python -m pip' to install. > > Tests with cmake are basically unfinished... The problem is ctest (the utility that cmake advises to use for testing and what add_test command generates data for) is pretty much unusable. Sure, you can add tests and execute them, it even supports parallel execution and running specific tests, but there are limitations. There is no easy way to pass arguments to the test programs and no setup action. Now as I rethink that, it's probably a good idea to add a script similar to win-tests.py that runs tests when built with cmake. It would be also nice to add itself to either generator target or to ctest. There is an issue that if we want it to be possible to run the tests via this script directly (from the root of the source tree or something like that) it needs to be aware of where cmake was configured which with out-of-tree build is any directory on the filesystem. This script could accept this directory as an argument which also defaults to something like 'out' or 'out/build' and could be potentially cached for convenience. It might also be a good idea to factor out the part that starts an http server from win-tests.py into a separate python file to reuse it here and maybe even merge with davautocheck.sh. I also wonder why we don't use python's native unittest framework [1]? Now it's probably too late to suggest that but rather as a crazy idea, how about we switch to it? My reasoning would be so we don't reinvent the wheel. The wheel that worked well enough for a long time, but still. By using a standard framework like this some IDEs might automatically discover tests and display them in the UIs. I think it's nice to have every test as a first class entity if it makes sense that "adapters" can load, run, and list in whatever way they want. I'm pretty sure that swig tests use this framework instead of svntest. Thoughts? [1] https://docs.python.org/3/library/unittest.html -- Timofei Zhakov
Re: Running tests with CMake
On 26. 5. 26 20:05, Branko Čibej wrote: I was looking at the test output of the CMake tests recently, and I noticed that they don't run XML validation even when it's not explicitly disabled. Looking at CMakeLists.txt, I found that each test program is run directly instead of through run_tests.py. I don't think that's a very good idea. Sure, developers can run individual tests like that, but run-tests.py adds some common semantics that the build system shouldn't just skip. Like, for example, running XML validation on command output. It also handles test parallelization. I understand that the CMake project can run each test function from all the tests separately, but it's possible to do that through run-tests.py. The alternative would probably be to install python-lxml and python-rnc2rng on the Windows test runners. I already did that for Linux runners (both autotools and cmake), because it is a bit more controlled than creating a virtual environment on the fly. However, I don't (want to?) know now to do that with vcpkg. Never mind. It was enough to run 'python -m pip' to install. -- Brane
Running tests with CMake
I was looking at the test output of the CMake tests recently, and I noticed that they don't run XML validation even when it's not explicitly disabled. Looking at CMakeLists.txt, I found that each test program is run directly instead of through run_tests.py. I don't think that's a very good idea. Sure, developers can run individual tests like that, but run-tests.py adds some common semantics that the build system shouldn't just skip. Like, for example, running XML validation on command output. It also handles test parallelization. I understand that the CMake project can run each test function from all the tests separately, but it's possible to do that through run-tests.py. The alternative would probably be to install python-lxml and python-rnc2rng on the Windows test runners. I already did that for Linux runners (both autotools and cmake), because it is a bit more controlled than creating a virtual environment on the fly. However, I don't (want to?) know now to do that with vcpkg. -- Brane
