I've tried submitting this as a launchpad answer but it expired
without further notice; I'm posting this here, I hope it's the right
place to discuss this.

I'm indirectly using zope.testrunner through zc.buildout zc.recipe.testrunner:

recipe = zc.recipe.testrunner
eggs = pydenji
defaults = [ "--auto-color", "-vv", "--tests-pattern", "^test_.*" ]

It works; but what happens if a test file matching the pattern has got
a serious failure, let's say an import or syntax error ?

Test-module import failures:

Module: pydenji.test.test_cmdline

line 13

SyntaxError: invalid syntax

Running tests at level 1
Running zope.testrunner.layer.UnitTests tests:
 Set up zope.testrunner.layer.UnitTests in 0.000 seconds.

[...] long test output [... ]

 Ran 73 tests with 0 failures and 0 errors in 0.188 seconds.
Tearing down left over layers:
 Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

Test-modules with import problems:

I can't show you such things here, but the "73" is green, which is the
colour for "all ok" - if any failure happens, the colors turn red,
while the "test modules with import problems" are just a tiny line
after that, and often gets overlooked.

Also, a test file matching the pattern but which not defines any test
is treated the very same way as a file with import problems, which is
probably not what it want.

The issues I can find are:

- I need to dig to the top in order to get the traceback; other
frameworks, like twisted's own trial, print all the tracebacks at the
bottom of the test run for easy debugging;
- test colors should not turn green if any test with import problem is
around; maybe an import/syntax error should count as a generic error.
- while an import issue is a serious fact - meaning the test can't be
run, and should be reported, a test module which does not define any
test could just issue a warning - it could just be a placeholder test,
or a not-yet-finished test case, and should not be a blocking issue.

Any thoughts on this?

Alan Franzoni
contact me at pub...@[mysurname].eu
Zope-Dev maillist  -  Zope-Dev@zope.org
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope )

Reply via email to