# from Ovid # on Thursday 04 January 2007 01:34 pm: >However, if you use the '-s' switch to shuffle your tests and bailout >is not first, then some tests will run until the BAIL_OUT is hit. > This seems to violate the principle that tests should be able to run > in any order without dependencies.
Is it possible to shuffle all but the first tests? The only reason I use BAIL_OUT is to give better diagnostics about something horribly wrong with the environment or some piece of code that won't compile. Example: http://svn.dotreader.com/svn/dotreader/trunk/t/ 01-load.t ensures that all of the modules compile -- I haven't forgotten a semicolon or missed a dependency. It uses checksums to optimize this (which, yes could be wrong, but I haven't seen it happen.) 01-db_timestamp.t is similar, but doesn't matter right now and could probably be handled by a build dependency when I get back to it. pretend you didn't see 03-prereq.t Other uses might include "You have no display" for GUI tests, "no network connection", "wrong operating system", "it is not 1965" or other diagnostics where you need to tell the user that something is very wrong without it getting buried (or scrolled away) by a pile of error messages. I do *prefer* the _gui tests to run first, but that's only a matter of learning about horribly wrong things earlier. Any of the tests in the directories could be run in any order, skipped around across directories, etc. In this case, the tests in the top directory are special. I suppose you would call them sanity tests. If they don't pass, you might as well go home. I suppose BAIL_OUT could be replaced with something like "if anything in 01-load.t fails, quit testing." I could imagine similar naming schemes in use with non-recursive t directories (probably with numbers (or low numbers) on the sanity tests and the rest un-numbered or maybe numbered 100+.) It seems unreasonable to call BAIL_OUT from anything but a sanity test, but I've been wrong before. Even in shuffle mode, the sanity tests should still run first, but what is the best way to explain that to the harness? --Eric -- Issues of control, repair, improvement, cost, or just plain understandability all come down strongly in favor of open source solutions to complex problems of any sort. --Robert G. Brown --------------------------------------------------- http://scratchcomputing.com ---------------------------------------------------
