On 15/01/2008, chromatic <[EMAIL PROTECTED]> wrote: > > On Monday 14 January 2008 15:42:49 Adam Kennedy wrote: > > > Test::Builder should just do what it's told. > > > > If the code that IS responsible for testing suites tells it to bail on > > fail, that's exactly what it should do. > > Let me rephrase this in an amusing way which demonstrates how silly this > idea > is. > > We have a program which runs suites of tests. It does this by spawning > new > processes which produce TAP somehow. Some of those processes produce TAP > by > using Perl modules which eventually use Test::Builder. Some do not. They > don't have to, because there's a separation between producing TAP and > consuming TAP, by design. > > This is why we have a program which runs suites of tests. > > Now some people say "If anything which generates TAP generates a TAP > failure, > I don't want to run the rest of the test suite." They have plenty of > potential reasons, and those reasons are reasonable reasons. > > We do have a program which runs suites of tests. Actually, we can have > several, because one of the goals of writing TAP::Parser which is now much > of > the guts of Test::Harness 3.0 is to allow people to write programs which > run > suites of TAP producers and consume and aggregate the resulting TAP. The > point is customized behavior. > > That's a digression though, because as I understand it the goal of putting > this OH NOES SOMETHING FALLDED!!! logic in Test::Builder is so that the > currently running test file (necessarily Perl, necessarily using > Test::Builder, so see the caveats about neither necessarily being true for > TAP generators) can stop running when it encounters a failure. > > Of course, that won't stop a whole test suite from continuing to run. Dot > dot > dot. > > ... unless you *could* figure out some way of writing a temporary file > somewhere that Test::Builder could check to see if it's supposed to abort > subsequent test files in the suite (caveats again applying about how they > have to be Perl programs which somehow use Test::Builder) and just not run > the tests. That's a little tricky though, if you use parallel test runs, > which has become one of the other goals of TAP::Parser, and I'm all for > that. > > I tried to think of a way to use environment variables set by one test > process > to communicate with another process, but then you've limited this to > working > only in certain shells on Windows for Perl programs which somehow use > Test::Builder. At least you can get away from the nasty file locking race > condition autoflush problem of using temporary files to communicate "Hey I > launched the process but ha ha just fooling not really why don't you die > now > thanks?" to child processes.
I think we might be mentally going about this wrong. A number of people thinking mostly about the practicalities of this seem to be thinking in terms of perfection, and really, perfection is not required. I'm increasingly thinking a common environment variable is the best communication mechanism. For the sake or argument lets call it PERL_TEST_FAINTING_GOAT Set the environment variable to true, and any components of the test process are instructed to abort and fall over as early as possible, with a methodology appropriate to that particular component. For Test::Builder, this is going to mean basically "bail on fail", with either an immediate bail or waiting until the next test (so as to assure diagnostics are in place). For the test harness, that means not running any more tests once a failure has been spotted, and either stopping any currently-running tests or ignoring any further results (in parallel mode). Since the entire purpose of the testing architecture we have is to allow diversity, perfection is more or less impossible. That's no reason not to treat this as a "do the best you can to abort" situation and encourage situation-specific behavior. Adam K
