I have wanted to add some testing to the CI and buildbot for some time,
but there are a few complications to consider. The first and simplest
barrier is that the current release of MacPorts errors out if you try to
run tests on a port that doesn't explicitly enable them. Related to that
is the fact that many (most?) ports don't have any meaningful tests to run.
There are unreleased changes that start to address this. I've added one
basic test that can be run for all ports, which at the moment just
checks if they are using `supported_archs noarch` appropriately. More
similarly simple and generic tests can be added, e.g. checking binaries
for broken linking. Ports that don't set `test.run yes` will run only
the built-in tests, while those that do will also run their custom tests
as defined in the Portfile.
The other problem is that some projects have, shall we say,
comprehensive test suites. This is undoubtedly a good thing for those
projects! But as a distro building thousands of ports, we have to have a
relatively tight limit on how long tests can take to run. There's
currently no way of differentiating a CI-appropriate test phase from an
unreasonably long-running one. Ideally we might have the option of
defining a short test phase that is run on CI, and a longer one that can
be run locally by maintainers and other interested parties if desired.
- Josh
