# from Smylers
# on Thursday 16 August 2007 11:40 pm:
>> I am certain that more than one 'extra tests' directory is needed,
>
>Why are you certain of this?
Because I already have a use for three 'profiles' in one project and I
can name a few others from elsewhere:
1. author (kwalitee, pod, etc)
2. gui
3. network
4. you must have an account/password on $external_service
5. postgres/mysql/whatever availability/setup/permissions
6. no modem attached to /dev/ttyS0
Thus: "more than one". What I'm saying is that "author" is not enough
(and/or not descriptive enough of a top-level directory of "extra
tests".)
> Anything under this directory isn't going to get run with 'normal'
> tests, which is an unusual requirement.
It's not *that* unusual in the context of CPAN. Anything that requires
intervention, setup, or access to a particular resource needs to be
skipped or bypassed during unattended pre-installation testing.
What I would ultimately like to see happen is that these subdirectories
fall into some form of standard(ish) naming scheme which would allow
testers and qa efforts to run particular subsets. That is, the smoke
box (or rigorous end-user upgrade verification process) could set
variables to test the author, 'gui', 'network', and 'postgres' tests[1]
without stumbling into e.g. "must have an X.com web-services account".
Which brings up a good point: I think "xt/foo.t" should be discouraged.
That is, if the tests in xt might assume any number of pre-configured
things, the only safe assumption is "test what you know you are
prepared to test".
Thus:
prove -rb xt/author
prove -rb xt/network
prove -rb xt/exhaustive
not `prove -rb xt/` -- which might crash for lack of gui before it gets
around to testing the network bit. Therefore, 'xt/foo.t' is a problem
because it has no category besides "it is extra".
[1] Yes, setup gets slightly tricky on the 'database', 'account', and
'devices' resources. I like to imagine that those could be fed
information via some standard mechanism such as a per-distro config
data-source -- the target usage there is more in the context of
distributed teams of developers and rigorous end-users than cpan
testers.
--Eric
--
The only thing that could save UNIX at this late date would be a new $30
shareware version that runs on an unexpanded Commodore 64.
--Don Lancaster (1991)
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
