Paul Cochrane wrote:
As a short term improvement of the situtation, how would a "make
test-cage" target sound? This would be the same as "make test" but
with the coding standards (and any other cage-cleanliness-checking
tests) shifted into this suite, and not run by default with "make
test".
In principle, I agree with this approach. It would represent a
particular application of the general principle I was advocating in my
YAPC::NA::2007 presentation: At any given point, run only the tests you
need to run then. There's no need to run tests to prove what you have
already accomplished.
More specifically, I would argue that the only tests that absolutely
*must* run when someone calls 'make test' are those which are needed to
demonstrate that:
* the Parrot executable created by 'make' can perform as demanded by the
specifications; and
* the tools we provide to accomplish the *remaining* steps (e.g., 'make
install', 'make clean') can accomplish those tasks.
By implication, there is no *absolute* need for 'make test' to include
tests which demonstrate that:
* the configuration tools are capable of building a Makefile;
* the build tools are capable of building an executable;
* the manifest tools are capable of writing a correct MANIFEST;
* the code meets our internally imposed coding standards; or
* anything else.
After all, if your configuration tools were not capable of building a
Makefile, you would never have gotten as far as 'make', much less 'make
test'. And if your build tools were incapable of meeting the demands
which 'make' places on them, you would never have gotten an executable
remotely capable of meeting the spec.
So I think it's not only possible to create specialized testing targets
which enable us to reduce the size of 'make test', it is desirable to do so.
HOWEVER, in practice things are never as simple as in principle. When I
refactored tools/build/pmc2c.pl, I wrote tests in t/tools/pmc2cutils/*.t
which are callable via 'make buildtools_tests'. The point of those
tests was to demonstrate that pmc2c.pl would be capable of doing what
'make' demanded of it *when 'make' did so*. So those tests were
structured to be run *before* 'make' and not included in 'make test'.
If you were to refactor the code underneath pmc2c.pl and then call 'make
buildtools_tests', you would catch bugs before 'make'. But if you
refactored that code and did *not* call 'make buildtools_tests' (or
'perl Configure.pl --test=build'), then your bugs would enter the Parrot
codebase because those tests are not included in 'make test'.
So, in principle, we can and should reduce the number of tests in 'make
test' and hence reduce its running time. But that requires *all* of us
-- not just ptc and me -- to be more diligent about running the
specialized testing targets before committing to trunk.
Can we organize ourselves to do so? That's a human problem, not a code
problem.
Thank you very much.
kid51
