[perl #47886] [TODO] 'make quicktest' target
The attached coretest.patch file adds a make coretest target to
Parrot. This target runs a smaller subset of tests than the normal
make test target. On my system, make test completes in ~290
seconds, while make coretest completes in ~200.
The following tests are not included in make coretest:
t/configure/
t/postconfigure/
t/compilers/json/
t/examples/
t/doc/
t/distro/
t/codingstd/
This patch also refactors the way that the default lists of test are
constructed in t/harness, so that we aren't globbing test directories
for lists of files we're not going to run anyway.
Lastly, this patch will make it possible for us to factor out the
$(RUNCORE_TESTS) variable that lists other core tests to be run for
the various runcores. At present adding core tests to Parrot involves
updating both t/harness and the Makefile, this refactor will make it
possible to keep the list in one place (t/harness).
Feedback welcome. If I hear support for this patch I'll go ahead and
apply it (with any suggested changes).
Thanks!
Pm
Index: t/harness
===
--- t/harness (revision 23180)
+++ t/harness (working copy)
@@ -102,6 +102,12 @@
my $gc_debug = grep { $_ eq '--gc-debug' } @ARGV;
@ARGV = grep { $_ ne '--gc-debug' } @ARGV;
+my $core_tests_only = grep { $_ eq '--core-tests' } @ARGV;
[EMAIL PROTECTED] = grep { $_ ne '--core-tests' } @ARGV;
+
+my $runcore_tests_only = grep { $_ eq '--runcore-tests' } @ARGV;
[EMAIL PROTECTED] = grep { $_ ne '--runcore-tests' } @ARGV;
+
my $html = grep { $_ eq '--html' } @ARGV;
@ARGV = grep { $_ ne '--html' } @ARGV;
@@ -134,6 +140,8 @@
-D[number] ... pass debug flags to parrot interpreter
--running-make-test
--gc-debug
+--core-tests
+--runcore-tests
--html
--tr ... run using Test::Run
EOF
@@ -148,54 +156,85 @@
$args =~ s/-D/-D$opts{D}/;
$args .= ' --gc-debug'if $gc_debug;
# XXX find better way for passing run_exec to Parrot::Test
-$args .= ' --run-exec'if $run_exec;
+$args .= ' --run-exec'if $run_exec;
$ENV{TEST_PROG_ARGS} = $args;
-# Pass in a list of tests to run on the command line, else run all the tests.
-my @default_tests = map {glob t/$_/*.t} qw(
-configure postconfigure compilers/imcc/* op pmc native_pbc dynpmc dynoplibs
-compilers/pge compilers/pge/p5regex compilers/pge/p6regex
-compilers/pge/perl6regex compilers/tge compilers/json
-library examples run src tools perl doc stm
+# Build the lists of tests to be run
+
+# runcore tests are always run.
+my @runcore_tests = qw(
+t/compilers/imcc/*/*.t
+t/op/*.t
+t/pmc/*.t
+t/native_pbc/*.t
+t/dynpmc/*.t
+t/dynoplibs/*.t
+t/compilers/pge/*.t
+t/compilers/pge/p5regex/*.t
+t/compilers/pge/perl6regex/*.t
+t/compilers/tge/*.t
+t/library/*.t
);
-# append the distribution checking tests to the default tests
-my @distro_tests = map { t/distro/$_ } qw(
-manifest.t
+# core tests are run unless --runcore-tests is present. Typically
+# this list and the list above are run in response to --core-tests
+my @core_tests = qw(
+t/run/*.t
+t/src/*.t
+t/tools/*.t
+t/perl/*.t
+t/stm/*.t
);
-push @default_tests, @distro_tests;
-# append the file_metadata.t only if we're running in a non-release checkout
-push @default_tests, 't/distro/file_metadata.t'
-if -e $Bin/../DEVELOPING;
-
-# collect the coding standard tests (that we want to run) together
-# append them to the list of default tests *only* if this is not a release
-my @coding_std_tests = map { t/codingstd/$_ } qw(
-c_code_coda.t
-c_header_guards.t
-c_indent.t
-c_struct.t
-check_toxxx.t
-copyright.t
-cppcomments.t
-cuddled_else.t
-filenames.t
-gmt_utc.t
-linelength.t
-pccmethod_deps.t
-pir_code_coda.t
-svn_id.t
-tabs.t
-trailing_space.t
+# configure tests are tests to be run at the beginning of 'make test';
+# standard tests are other tests run by default with no core options
+# present
+my @configure_tests = qw( t/configure/*.t t/postconfigure/*.t );
+my @standard_tests = qw(
+t/compilers/json/*.t
+t/examples/*.t
+t/doc/*.t
+t/distro/manifest.t
);
-# XXX: This takes WAY too long to run: perlcritic.t
-push @default_tests, @coding_std_tests
-if -e $Bin/../DEVELOPING;
+# add metadata.t and coding standards tests only if we're not DEVELOPING
+unless ( -e $Bin/../DEVELOPING ) {
+push @standard_tests, 't/distro/metadata.t';
+push @standard_tests, map { t/codingstd/$_ } qw(
+c_code_coda.t
+c_header_guards.t
+c_indent.t
+c_struct.t
+check_toxxx.t
+copyright.t
+cppcomments.t
+cuddled_else.t
+filenames.t
+gmt_utc.t
+linelength.t
+pccmethod_deps.t
+pir_code_coda.t
+svn_id.t
+tabs.t
+trailing_space.t
+);
+# XXX: This takes WAY too long to
Re: [perl #47886] [TODO] 'make quicktest' target
On Wednesday 28 November 2007 09:19:51 Patrick R. Michaud via RT wrote: The attached coretest.patch file adds a make coretest target to Parrot. This target runs a smaller subset of tests than the normal make test target. On my system, make test completes in ~290 seconds, while make coretest completes in ~200. +1 -- c
Re: [perl #47886] [TODO] 'make quicktest' target
On 28/11/2007, Patrick R. Michaud via RT [EMAIL PROTECTED] wrote: The attached coretest.patch file adds a make coretest target to Parrot. This target runs a smaller subset of tests than the normal make test target. On my system, make test completes in ~290 seconds, while make coretest completes in ~200. The following tests are not included in make coretest: t/configure/ t/postconfigure/ t/compilers/json/ t/examples/ t/doc/ t/distro/ t/codingstd/ This patch also refactors the way that the default lists of test are constructed in t/harness, so that we aren't globbing test directories for lists of files we're not going to run anyway. Lastly, this patch will make it possible for us to factor out the $(RUNCORE_TESTS) variable that lists other core tests to be run for the various runcores. At present adding core tests to Parrot involves updating both t/harness and the Makefile, this refactor will make it possible to keep the list in one place (t/harness). Feedback welcome. If I hear support for this patch I'll go ahead and apply it (with any suggested changes). Thanks! Pm +1
[perl #47886] [TODO] 'make quicktest' target
Committed patch in r23209, including refactoring $(RUNCORE_TEST_FILES) out of the root Makefile and into t/harness. Pm
[perl #47886] [TODO] 'make quicktest' target
# New Ticket Created by Patrick R. Michaud # Please include the string: [perl #47886] # in the subject line of all future correspondence about this issue. # URL: http://rt.perl.org/rt3/Ticket/Display.html?id=47886 Earlier today on irc #parrot, Coke, allison, barney, and I were discussing ways to speed up testing for developers -- especially things like the codingstd tests that are currently run with every 'make test'. After much discussion, we decided we'd like to have a make quicktest target that runs a subset of core Parrot tests that verify that the functionality is intact. Code developers could use this target to quickly verify that the changes they're making to code aren't causing functional features to fail. The existing 'make test' target would remain like it is now, running a fuller suite of tests that verify that coding standards are met, file properties are correct, examples are still running, etc. Thus, someone making iterative core changes for Parrot can use make quicktest to make sure functional features are still correct, and then use make test to get the standard set of tests prior to a commit. We discussed the possibility of the reverse -- i.e., have make test be the smaller set of tests and use a different target for testing prior to a commit, but we decided that it makes more sense for make test to be the standard suite and make quicktest to be the just make sure everything works. T. Improvements like this would also be very welcome -- but there still seems to be value in having a 'make quicktest' target available. My suggestion would be that 'make quicktest' would run the core tests (e.g., op, pmc, oo), the compilers/ tests (imcc, pge, tge), and library/ tests. This means that quicktest would avoid the configure, postconfigure, examples, tools, perl, doc, distro, and codingstd tests that are part of 'make test'. Thanks! Pm
Re: [perl #47886] [TODO] 'make quicktest' target
On Tue, Nov 27, 2007 at 02:01:25PM -0800, Patrick R.Michaud wrote: After much discussion, we decided we'd like to have a make quicktest target that runs a subset of core Parrot tests that verify that the functionality is intact. FWIW, this target is known as coretest in Perl 5. -- Paul Johnson - [EMAIL PROTECTED] http://www.pjcj.net
