Might it make sense for a CI server based execution of the smoke tester to not actually run the tests, instead leaving that to other build jobs? I ask because it takes longer to deduce a smoke tester failure than it does for a regular build (massive console log). Regular builds highlight the test failures up front. It's not a big deal, and admittedly if our tests didn't fail too much, this wouldn't be so much a problem. Also if a CI run of the smoke tester didn't run the tests, presumably it would run very quickly and alert us to the other issues, like precommit problems. So a smoke tester failure would be more indicative of the nature of a problem (categorically not a test failure -- something else), leaving tests to separate jobs. Does this make sense to folks?
Cheers, ~ David -- Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com
