In many situations my test suites are such that if more than one test
fails, they typically provide more information together than just their
individual results ("the total is greater than the sum of the parts", if
you will), so it takes less time for me to track down the actual problem if
I see all tests that fail, rather than just the first one. This is not
currently the default behavior of the macros in Base.Test:
using Base.Test
function throws_error()
error("oops")
end
@test 2 == 3
@test 2 == 2
@test 4 < 3
@test throws_error()
Output:
ERROR: test failed: 2 == 3
in error at error.jl:21
in default_handler at test.jl:19
in do_test at test.jl:39
in include at boot.jl:244
while loading [...]/example.jl, in expression starting on line 7
Note that no information is given about the third test.
I suppose this could be remedies using a custom handler, similar to this
example in the docs<http://docs.julialang.org/en/latest/stdlib/test/#handlers>,
so I started building one. Running tests with this handler is much more
informative, in some aspects, but much less in others:
using Base.Test
nonfatal_handler(r::Test.Success) = Test.default_handler(r)
function nonfatal_handler(r::Test.Failure)
print_with_color(:red, "Test FAILED: $(r.expr)\n")
end
function nonfatal_handler(r::Test.Error)
print_with_color(:red, "ERROR during test: $(r.expr)\n")
showerror(STDOUT, r.err, r.backtrace)
end
function throws_error()
error("oops")
end
Test.with_handler(nonfatal_handler) do
@test 2 == 3
@test 2 == 2
@test 4 < 3
@test throws_error()
end
Output:
Test FAILED: 2 == 3
Test FAILED: 4 < 3
ERROR during test: throws_error()
oops
in throws_error at [...]/example.jl:13
Now I see all failures, and not just the first one - but I get much less
information about them. In particular, I don't even get a line number in
the test file for test cases that fail but don't throw errors. I'm also not
entirely very about hard-coding STDOUT in the error handler, but I couldn't
figure out a way around it without interrupting control flow.
This isn't perfect, though - if we re-order the tests, so that the throwing
case is in the middle, control flow is interrupted anyway, and the tests
after don't run. Can this even be fixed without altering the @test macro(s)?
Any suggestions on how to improve this? And if this is improved enough to
give substantial information about each case, without interrupting flow,
would it be an interesting addition to the library?
Best regards
// Tomas
