Adam,
I have one more edgey case I'd like to see on this list:
>
> 13. Tests exist, but fail to be executed.
> There is tests, but the tests themselves aren't failing.
> It's the build-process that is failing.
>
> 14. Tests run, and some/all tests fail.
> The normal FAIL case due to test failures.
>
> 15. Tests run, but hang.
> Covers "non-skippable interactive question".
> Covers infinite attempts to connect network sockets.
> Covers other such cases, if detectable.
Tests run, but >50% (or maybe >80%?) are skipped.
From what I've seen, the most common cause of this is that the
package is untestable with the current build configuration. Eg; you needed
to specify a webserver or database or something to get these tests to run.
Apache::Test provides some hooks for autotesters to configure themselves to
test packages using it, IMHO setting DBI_DSN etc should be enough for
packages that test against a database.
I've been thinking a lot about how to properly autotest
database-driven packages lately. So far the best I've got is:
- If a package depends on a particular DBD:: driver, and you have
the matching database server available, specify a DBI_DSN that uses that
driver.
- If a package depends on DBI, but no particular driver, try testing
it with every driver you can use, starting with SQLite since it does not
need a "live" database server.
In the case where a package supports multiple, but not all, database
driver backends, they would probably depend on DBI, but "reccomend" each
DBD:: driver in the META.yml, which would be covered by the first bullet.
Cheers,
Tyler
