Moritz Lenz wrote:
So I'd like to hear your opinions: do you think
adverb-based testing is a good idea? If you don't like
it, do you see any other good way to tackle the
problems I mentioned above?

After reading everything in this thread to date and in order to structure my thoughts, I wrote up some assertions a suggestion.

1) A perl6 implementation is to be certified by its ability to pass the test suite. The test functionality is implemented in the implementation (the implicit recursion has been noted in other threads). This means the behaviour of the test functionality must be specified and verifiable.

Hence the assertion that perl6 is to be specified by a suite of documents and tests seems to me to imply that test functionality must be an inherent part of the language specification.

2) Test functionality for the compiler must be as simple to implement as possible, so that it can be incorporated into the implementation at an early stage.

3) Although test functionality for more complex software, eg., event-driven GUIs, could be constructed from simple specified test functionality, more complex forms will be developed to isolate the features that need testing and to provide the diagnostics.

In other words, test functionality sufficient for the compiler may not be adequate for module testing. But other functions can be developed in Test modules that can be hooked into a general testing approach.

4) Testing software is different from debugging or running software. Running is about providing functionality to the user. Debugging is about getting expected behaviour and discovering why behaviour exhibited is not what is expected / specified. Testing is about demonstrating that the functionality provided is the functionality expected / specified under all specified conditions.

It seems to me that the ethos of testing could be much wider than just a part of the development stage. From a risk-management perspective, mission-critical software (especially when it is complex and large) should be tested regularly against a standard test suite, because random errors may occur in the software (eg., a power surge subtly corrupts the contents of hard disk storage), and particularly after any upgrade of hardware or ancillary software, or any other environmental change, to say nothing of changes (upgrades) in the software itself.

Indeed, the inclusion of test functionality in the language and the focus on test suites as part of perl6 culture would make software written in perl6 extremely desirable in risk-sensitive companies.

Consequently, it seems to me that the following might be useful:

a) a global variable $*TESTING which defaults to FALSE (or should it be $?TESTING ?)

It could be set lexically so that specific software / modules can be tested without triggering tests in other used modules.

b) When $*TESTING is TRUE, any TEST block is executed.

c) Within a TEST block, the tenary <?? !!> is defined slightly differently, thus for <perl6 boolean expression> ?? <diagnostic if expression is TRUE> !! <diagnostic if expression is FALSE>;

<perl6 boolean expression> is guaranteed to return Boolean::FALSE if an exception or failure condition is encountered when evaluating it.

Some advantages of this approach over :OK<>:
- no new behaviour outside of a TEST block is defined, no change to adverbs or boolean operators. - any expression that leads to a boolean result (note that :OK is suggested to be defined only on boolean operators) can be included in the expression, eg., an entire block. - The programer has control over both the "True" diagnostic, as in the :OK<> syntax, but also over the 'False' diagnostic, thus allowing a degree of introspection on the component of the expression, which the programmer has more knowledge about than the compiler. - Since the variables used in the boolean expression are available to the programmer for both diagnostics, there is no need for special magic to generate the failure diagnostic, which seems to be the situation with :OK<>. - Since it is the programmer that defines the False diagnostic, no extra autogenerated macros are needed. - The minimum that is needed for a test would be to specify a 'true' diagnostic and the $! error variable, eg.,
TEST { 2 == 2 ?? say 'constants are constants' !! say $! };

d) A TEST block is specified to react to exceptions / failures in a different manner than in normal blocks. Uncaught exceptions are discarded at the end of a block. Thus compiler / module / software failures do not stop the software from continuing, unless specifically required by the programmer to do so within the block.

e) Other functions that are useful in test suites, such as plan, could be defined later as wrappers around "?? !!"


Reply via email to