Re: skipping and installing
Geoffrey Young wrote: bugger. if the bloody package continues running through the rest of the tests even after encountering a failure -- which is right, and what it does -- it should use a uniform report syntax and most certainly *not* conceal details about the totals.. :-( I can't say that I follow all of that :) i mean that some applications halt as soon as they encounter the first error. the test suite, on the other hand, doggedly continues until it has tried all the tests. which is, of course, how it should be. but its summary information should not include totals when all succeed and omit them when there are failures. but I ran a few tests with my own A-T modules and see that I do indeed get the final summary line all the time - both when there are failures and when there are not. so, it looks like an issue with your wrapper and not A-T core, yes? no. if i run only a test script that passes completely, i get t/modules/accessok All tests successful. Files=1, Tests=408, 4 wallclock secs ( 3.13 cusr + 0.43 csys = 3.56 CPU) if i run only a script that fails, i get t/apache/limitsFAILED test 9 Failed 1/10 tests, 90.00% okay Failed Test Stat Estat Total Fail Failed List of Failed --- t/apache/limits.t 101 10.00% 9 and that's it. btw, did t/TEST -conf work for you? yes, that was the ticket. -- #kenP-)} Ken Coar, Sanagendamgagwedweinini http://Ken.Coar.Org/ Author, developer, opinionist http://Apache-Server.Com/ Millennium hand and shrimp!
Re: skipping and installing
so, it looks like an issue with your wrapper and not A-T core, yes? no. if i run only a test script that passes completely, i get t/modules/accessok All tests successful. Files=1, Tests=408, 4 wallclock secs ( 3.13 cusr + 0.43 csys = 3.56 CPU) if i run only a script that fails, i get t/apache/limitsFAILED test 9 Failed 1/10 tests, 90.00% okay Failed Test Stat Estat Total Fail Failed List of Failed --- t/apache/limits.t 101 10.00% 9 and that's it. ok, I can reproduce that. don't know what the fix is, though. btw, did t/TEST -conf work for you? yes, that was the ticket. cool. --Geoff
Re: skipping and installing
Geoffrey Young wrote: hmm, I get the following with 5.8.3 t/protocol/nntp-like...ok t/ssl/all..skipped all skipped: cannot find module 'mod_ssl', cannot find module 'LWP::Protocol::https' All tests successful, 7 tests and 14 subtests skipped. Files=42, Tests=2355, 68 wallclock secs (48.61 cusr + 3.88 csys = 52.49 CPU) i get: t/modules/status..ok t/modules/vhost_alias.ok Failed Test Stat Wstat Total Fail Failed List of Failed --- t/apache/chunkinput.t 92 22.22% 7 9 t/apache/contentlength.t 207 35.00% 2 6 10 14 16 18 20 t/apache/limits.t 101 10.00% 9 t/modules/autoindex2.t 32 66.67% 1 3 t/modules/expires.t924 4.35% 13 21 33 45 14 subtests skipped. and that's it. are you using an older version of perl perhaps? yep, 5.6.1. why would that make a difference? newer versions of the test modules? -- #kenP-)} Ken Coar, Sanagendamgagwedweinini http://Ken.Coar.Org/ Author, developer, opinionist http://Apache-Server.Com/ Millennium hand and shrimp!
Re: skipping and installing
t/modules/expires.t924 4.35% 13 21 33 45 14 subtests skipped. and that's it. oh, I see what you mean, you're missing the Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay. line. are you using an older version of perl perhaps? yep, 5.6.1. why would that make a difference? newer versions of the test modules? yes, Apache-Test relies on Test.pm, Test::Harness, etc for most of the heavy lifting, so the nuances of reporting is driven more by them than Apache-Test. however, in this case it may be A-T's fault. a quick test shows that even with 5.6.1 the missing line shows up for non-A-T tests. furthermore, it's sent to STDERR whereas the results you show go to STDOUT. maybe that's the problem, STDERR is whacked someplace? I can't see where, but who knows. anyway, I'll look at it soonish unless someone beats me to it. btw, the t/SKIP fix was checked in yesterday if you didn't see it. thanks! --Geoff
Re: skipping and installing
Geoffrey Young wrote: t/modules/expires.t924 4.35% 13 21 33 45 14 subtests skipped. and that's it. oh, I see what you mean, you're missing the Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay. line. actuall, what i'm missing (and wanting) is the All tests successful, 7 tests and 14 subtests skipped. Files=42, Tests=2355, 68 wallclock secs (48.61 cusr + 3.88 csys = 52.49 CPU) lines. (unless the 'subtests skipped' line is respresented by my '14 subtests skipped' line.) the total of files and tests is very important for my application. however, in this case it may be A-T's fault. a quick test shows that even with 5.6.1 the missing line shows up for non-A-T tests. furthermore, it's sent to STDERR whereas the results you show go to STDOUT. maybe that's the problem, STDERR is whacked someplace? I can't see where, but who knows. in my case, it's possibly my wrapper that's gobbling the line. for pity's sake, why it that going to stderr when everything else is going to stdout? erk. now i'm seeing it, but *only* if all of the tests are successful. i need to see it regardless, and including the successful/unsuccessful counts. oh, bogus. it looks like that's in Test/Harness.pm -- and it won't get displayed if there are *any* failures. bugger. if the bloody package continues running through the rest of the tests even after encountering a failure -- which is right, and what it does -- it should use a uniform report syntax and most certainly *not* conceal details about the totals.. :-( -- #kenP-)} Ken Coar, Sanagendamgagwedweinini http://Ken.Coar.Org/ Author, developer, opinionist http://Apache-Server.Com/ Millennium hand and shrimp!
Re: skipping and installing
in my case, it's possibly my wrapper that's gobbling the line. for pity's sake, why it that going to stderr when everything else is going to stdout? erk. now i'm seeing it, but *only* if all of the tests are successful. i need to see it regardless, and including the successful/unsuccessful counts. oh, bogus. it looks like that's in Test/Harness.pm -- and it won't get displayed if there are *any* failures. bugger. if the bloody package continues running through the rest of the tests even after encountering a failure -- which is right, and what it does -- it should use a uniform report syntax and most certainly *not* conceal details about the totals.. :-( I can't say that I follow all of that :) but I ran a few tests with my own A-T modules and see that I do indeed get the final summary line all the time - both when there are failures and when there are not. so, it looks like an issue with your wrapper and not A-T core, yes? btw, did t/TEST -conf work for you? --Geoff
Re: skipping and installing
Rodent of Unusual Size wrote: 2. t/SKIP (which i haven't used for a long time) seems to have changed behaviour. for instance, t/php runs regardless of whether i have 'php' or 't/php' or anything else mentioning it in t/SKIP. what is the method nowadays to get a test script completely ignored (so it doesn't even say 'skipped' in the output)? okey, i've found the cause of the first part -- namely, it isn't t/SKIP any more, it's ./SKIP . this appears to be a regression introduced in rev 1.13 (HEAD) when the chdir to t/ was removed. the skip file is still accessed as 'SKIP', but the current directory context has changed. it looks as though the attached patch will fix this regression.. but is it the appropriate solution? my other questions remain.. -- #kenP-)} Ken Coar, Sanagendamgagwedweinini http://Golux.Com/coar/ Author, developer, opinionist http://Apache-Server.Com/ Millennium hand and shrimp!
Re: skipping and installing
Rodent of Unusual Size wrote:
it looks as though the attached patch will fix this regression.. but
is it the appropriate solution?
der! would help if i attached it, wouldn't it..
--
#kenP-)}
Ken Coar, Sanagendamgagwedweinini http://Golux.Com/coar/
Author, developer, opinionist http://Apache-Server.Com/
Millennium hand and shrimp!
Index: Apache-Test/lib/Apache/TestHarness.pm
===
RCS file:
/home/cvs/httpd-test/perl-framework/Apache-Test/lib/Apache/TestHarness.pm,v
retrieving revision 1.13
diff -u -r1.13 TestHarness.pm
--- Apache-Test/lib/Apache/TestHarness.pm 31 Jan 2004 01:12:17 -
1.13
+++ Apache-Test/lib/Apache/TestHarness.pm 23 Feb 2004 17:29:27 -
@@ -23,7 +23,7 @@
#skip tests listed in t/SKIP
sub skip {
my($self, $file) = @_;
-$file ||= 'SKIP';
+$file ||= 't/SKIP';
return unless -e $file;
Re: skipping and installing
-$file ||= 'SKIP';
+$file ||= 't/SKIP';
excellent, thanks for tracking that down.
we just need to be a bit more platform independent. if you could verify
that the attached patch works ok for you, I'll commit it.
--Geoff
Index: lib/Apache/TestHarness.pm
===
RCS file:
/home/cvspublic/httpd-test/perl-framework/Apache-Test/lib/Apache/TestHarness.pm,v
retrieving revision 1.13
diff -u -r1.13 TestHarness.pm
--- lib/Apache/TestHarness.pm 31 Jan 2004 01:12:17 - 1.13
+++ lib/Apache/TestHarness.pm 23 Feb 2004 17:40:32 -
@@ -23,7 +23,7 @@
#skip tests listed in t/SKIP
sub skip {
my($self, $file) = @_;
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
return unless -e $file;
Re: skipping and installing
1. is it possible to run the suite *without* doing a 'make install'? in my development environment i discovered that i had to do that in order oget rid of some cruft left behind from having done a 'make install' in the past. however, i'd prefer to not have to force the one-buttoners to have to install the modules system-wide. t/TEST -conf ought to do it I think. 3. after the suite has been run, i notice that there isn't (any longer) a grand total line (x tests run, y failures). is that info accessible somehow? hmm, I get the following with 5.8.3 t/protocol/nntp-like...ok t/ssl/all..skipped all skipped: cannot find module 'mod_ssl', cannot find module 'LWP::Protocol::https' All tests successful, 7 tests and 14 subtests skipped. Files=42, Tests=2355, 68 wallclock secs (48.61 cusr + 3.88 csys = 52.49 CPU) are you using an older version of perl perhaps? HTH --Geoff
Re: skipping and installing
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
oops. forgot to add the class up top. guess I'm not quite with it today yet :)
--Geoff
Index: lib/Apache/TestHarness.pm
===
RCS file:
/home/cvspublic/httpd-test/perl-framework/Apache-Test/lib/Apache/TestHarness.pm,v
retrieving revision 1.13
diff -u -r1.13 TestHarness.pm
--- lib/Apache/TestHarness.pm 31 Jan 2004 01:12:17 - 1.13
+++ lib/Apache/TestHarness.pm 23 Feb 2004 17:51:28 -
@@ -4,6 +4,7 @@
use warnings FATAL = 'all';
use Test::Harness ();
+use Apache::Test ();
use Apache::TestSort ();
use Apache::TestTrace;
use File::Spec::Functions qw(catfile catdir);
@@ -23,7 +24,7 @@
#skip tests listed in t/SKIP
sub skip {
my($self, $file) = @_;
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
return unless -e $file;
Re: skipping and installing
Geoffrey Young wrote:
-$file ||= 'SKIP';
+$file ||= 't/SKIP';
excellent, thanks for tracking that down.
we just need to be a bit more platform independent. if you could verify
that the attached patch works ok for you, I'll commit it.
Index: lib/Apache/TestHarness.pm
===
RCS file:
/home/cvspublic/httpd-test/perl-framework/Apache-Test/lib/Apache/TestHarness.pm,v
retrieving revision 1.13
diff -u -r1.13 TestHarness.pm
--- lib/Apache/TestHarness.pm 31 Jan 2004 01:12:17 - 1.13
+++ lib/Apache/TestHarness.pm 23 Feb 2004 17:40:32 -
@@ -23,7 +23,7 @@
#skip tests listed in t/SKIP
sub skip {
my($self, $file) = @_;
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
Geoff, you are making a good point of removing the hardcoding of t/ towards
the idea of being able to split the test suite. There are a few other places
where t/ is hardcoded. But at those places, e.g.:
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg;
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg.t;
Apache::Test::vars('serverroot') is not available yet. So perhaps we need to
deduct the serverroot directory from the location of the driving script. I'd
think in the recently added code in TestRun.pm:
sub run {
my $self = shift;
# assuming that test files are always in the same directory as the
# driving script, make it possible to run the test suite from any place
# use a full path, which will work after chdir (e.g. ./TEST)
$0 = File::Spec-rel2abs($0);
if (-e $0) {
my $top = dirname dirname $0;
chdir $top if $top and -d $top;
}
So may be it should be:
my $server_root;
my $project_root;
sub server_root { $server_root }
sub project_root { $project_root }
sub run {
my $self = shift;
# assuming that test files are always in the same directory as the
# driving script, make it possible to run the test suite from any place
# use a full path, which will work after chdir (e.g. ./TEST)
$0 = File::Spec-rel2abs($0);
if (-e $0) {
$server_root = dirname $0;
$project_root = dirname $server_root;
chdir $project_root if $project_root and -d $project_root;
}
and now server_root() and project_root() are available across the A-T project.
That's of course if all projects will always put TEST (or its equivalent) in
t/ or another directory.
__
Stas BekmanJAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide --- http://perl.apache.org
mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: skipping and installing
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
Geoff, you are making a good point of removing the hardcoding of t/
towards the idea of being able to split the test suite.
well, it's an idea that we're all working toward, yourself included :)
There are a few
other places where t/ is hardcoded. But at those places, e.g.:
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg;
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg.t;
Apache::Test::vars('serverroot') is not available yet.
are you sure? at the top of that function it grabs
my $top_dir = $self-{test_config}-{vars}-{top_dir};
surely if top_dir is available then serverroot is also, since they're both
setup in TestConfig::new().
what I would change there is this
#need the t/ for stat-ing, but don't want to include it in test output
$arg =~ [EMAIL PROTECTED](?:\./)?t/@@;
so that t/ is derived from the proper serverroot, then those two lines can
concat the proper value back.
but this is also an aside from the patch I posted, since it seems to be ok
(now that I've tested it :)
or maybe I'm not reading you right. it's monday and I'm not all here :)
--Geoff
Re: skipping and installing
Geoffrey Young wrote:
-$file ||= 'SKIP';
+$file ||= catfile Apache::Test::vars('serverroot'), 'SKIP';
Geoff, you are making a good point of removing the hardcoding of t/
towards the idea of being able to split the test suite.
well, it's an idea that we're all working toward, yourself included :)
;)
There are a few
other places where t/ is hardcoded. But at those places, e.g.:
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg;
Apache-Test/lib/Apache/TestRun.pm:push @tests, t/$arg.t;
Apache::Test::vars('serverroot') is not available yet.
are you sure? at the top of that function it grabs
my $top_dir = $self-{test_config}-{vars}-{top_dir};
surely if top_dir is available then serverroot is also, since they're both
setup in TestConfig::new().
I think you are right.
what I would change there is this
#need the t/ for stat-ing, but don't want to include it in test output
$arg =~ [EMAIL PROTECTED](?:\./)?t/@@;
so that t/ is derived from the proper serverroot, then those two lines can
concat the proper value back.
Actually that comment is not correct any longer. We now include t/ in the
output if you have noticed. Because otherwise T-H won't find those tests. So
it probably should be adjusted.
but this is also an aside from the patch I posted, since it seems to be ok
(now that I've tested it :)
or maybe I'm not reading you right. it's monday and I'm not all here :)
I think you can commit your patch right away. It looks right to me.
__
Stas BekmanJAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide --- http://perl.apache.org
mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
