On Tue, Jan 29, 2013 at 11:38 AM, Nick Wellnhofer <[email protected]> wrote:
> Another nice feature of TAP is that you have to split your tests into > separate programs. So if one of the test programs crashes, the others are > not affected. +1 > I thought about using TAP for testing the C bindings, but we'd also have to > provide a TAP consumer like Perl's 'prove' utility to interpret the tests. Sounds good. My impression is that writing a test runner is pretty involved in comparison to writing a TAP producer library. > For now, I decided to go with pluggable formatters for the test output and > provide a TAP formatter and a custom Clownfish formatter. The latter can be > used to print a test summary from the test program directly. It will be great if the pluggable formatter is all the scaffolding we need to support all target platforms. :) >> One thing I've wanted to do for a while is have our C tests return true on >> success, false on failure, and then use that return value to determine the >> exit value for the test script. Right now our C-based test scripts still >> exit >> success even when some tests fail. > > I just pushed a commit with some major reworks which also address this > issue. Nick++ > Another thing I'd like to have is some kind of central registry for > tests, so we don't have to maintain code like the following in every host > language binding: > > http://s.apache.org/n3 That silly registry is only necessary because all our Lucy test classes are inert. If we make Lucy::Test::TestBatch subclassable[1] and have Lucy's test classes subclass it, then we can use the following incantation: my $test_batch = Lucy::Test::Analysis::TestNormalizer->new( format => 'tap', ); my $success = $test_batch->run_tests; exit($success ? 0 : 1); I'd also like to propagate the work we did with Charmonizer's test harness a while back to Clownfish/Lucy, and make the test macros operate on a hidden global TestBatch... #define CFISH_TEST_OK(_expression, _message) \ CFTest_TestBatch_Test_True(cfish_TestBatch_current, (_expression), \ (_message)) ... allowing us to make changes like this: - TEST_FALSE(batch, - Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]), + TEST_FALSE(Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]), "Equals() false with different normalization form"); Using a hidden global works fine until you need to test threads -- at which point your tests can use full method invocations on local TestBatch objects. Marvin Humphrey [1] Eventually, that should be Clownfish::Test::TestBatch.
