On 21/08/2016 9:02 PM, Solomon E wrote:
On Saturday, 20 August 2016 at 20:39:13 UTC, Engine Machine wrote:
We have a unittest, what about an examples?
Examples are self contained short programs, each block acts as a
"main" function. One can run all the examples and spit out all the
output consecutively. It also allows for more robust testing since it
adds another layer.
It would provide better examples for docs also. Instead of using
assert to assert stuff we can see a real program in action. The output
of the example could easily be generated and added to the end of the
code.
Seems like a win win! (maybe examples is not a good keyword, but that
is moot)
It seems like there could be a library function that
compile-time-reflects to make a collection of all the functions in a
module that have names starting with "maintest" and calls each of them
in a try block, with a catch block that just prints the error messages
to stderr and incorrect return codes and counts the total fails and
finally counts the number of tests run to return the success ratio.
That would be trivial to write in a language that has runtime reflection
like Python. Probably someone who knows D well could write it for D in a
few minutes. It's too hot here (in the Seattle area) right now for me
with my limited D knowledge to feel like trying it instead of kibitzing.
The point would be when writing something that has one development-stage
'main', other mains that could be used instead or are planned to be used
or are just maintests would be live all the time (as long as the
function that calls them as tests is included in the real main or, as it
should probably be, in a non-release version block or in a unittest block.)
A well-featured interface would allow calling maintests with simulated
command line args specific for each one, or a list of lists of strings
times a list of functions, args[][] times maintest*[], for more thorough
testing. I've used a feature in NetBeans for Java (hobby project, not a
pro dev here) that calls main for a configured build with stored args,
but it required manual GUI interaction to switch between the builds.
Also a good feature would allow a maintest "success" to be a user
specified value of a return code for certain args, because when writing
something that's supposed to return specific error codes, that
functionality should be tested.
hypothetical usage code:
unittest
{
auto mainResults = Maintests([1: [["-x", "-y"], ["0"]], 2: [["-x",
"-A"], ["3"]]);
// -x and -A switches are incompatible args, to return 3 as an
arbitrary error code
assert(mainResults.fails == 1 && mainResults.tests["X"][2].success
== false);
}
int maintestX (string[] args)
{
// TODO
return 0; // it's going to fail the second test, on purpose
}
int maintestY (string[] args)
{
// TODO
return 3 * (args[2] == "-A"); // laziest way to pass current tests
}
result:
4 tests run with each unittested release build, which would otherwise
require two additional versions of the module containing main to be
compiled and run with specific args 2 times each, with the correct
return values differing for extra difficulty, or else require
assert(maintestX(["-x", "-y"]) == 0); etc. to be written X4 with
variations. The number of variations to write out manually would
increase by the number more test cases times the number of variations of
main, and wouldn't ensure that functions that look like "maintest" are
tested.
// ? void maintestZ () { }
// would fail a test that specifies any return value other than 0
// ? int maintest() { throw new Exception("no catch"); }
// would always fail
This might be generalized to
auto results = TestPattern!("maintest*", int[string[]])([["-a"]: 0]);
which might have some more general uses.
[I wrote the above, then I felt like, nah, I don't want to post things
that sound like asking for other people to do more work. Now that I've
done something on it myself, I'm posting the above for documentation.]
Writing your own test runner isn't hard[0]. Although I'm not entirely
sure this is even on the right continent as for purpose or scope.
Either way while there might be a need here, this is going about it all
wrong. After all, the documentation we do currently generate is for
libraries not for whole programs which this is destined for and you're
gonna wanna do it kinda custom as per the documentation itself.
[0] https://github.com/dlang/druntime/blob/master/src/test_runner.d
