At least in our test suites, any test which fails needs to have an open bug against it.
Any test which fails AND has an open bug against it should be considered as "passed" by any
test harness with a positive IQ.
I'm not too familiar with the JUnit architecture, but the should be some way to configure an exclude list of all tests
which fail due to logged bugs.
I'd seriously recommend against ifdeffing all your code with functionality which should be handled by the harness instead of placing
all the login in the test itself.
my 0.2e
WH
Glen Daniels wrote:
What do people think about a system property switch we can use to activate/deactivate tests which are "bleeding edge"? Right now, we have tests (like TestCircularRefs and test/wsdl/gateway) which are in the tree but test things which are currently known to be broken. The way we deal with this at present is to comment out the broken stuff so the tests run.
How about instead, we surround "sketchy" things which SHOULD pass but don't yet with
if (bleedingEdge) {
}
where bleedingEdge gets set by running with -Daxis.bleedingEdge on the command line.
That way people interested in working on making those tests run won't have to uncomment/recomment various bits and pieces. Once the test works, just remove the if.
Opinions?
--Glen
