>>> Alan also suggested against integrating the test suite as-is, because as he >>> said, "Remember, don't run these tests on a production system. They WILL >>> cause panics and deadlocks, and they may cause data loss too.” >>> >>> Cheers, >>> -Garrett >> >> Wait, we want to sweep those bugs under the rug? What exactly is wrong with >> making a test harness that can very easily reproduce a known problem? The >> chances are that anyone will dive into it once the bug is easily >> reproducible. > > Sweeping bugs under the rug is not what I plan on doing; I’m marking > these as expected failures, as opposed to having them continually panic a > machine. Once a ZFS dev takes a look at the issue and resolves them, then the > ZFS dev can remove the “bail” calls I’m adding to the testcases. > Cheers, > -Garrett
Yes, disabling tests that fail leads to an ineffectual test suite. A test suite that never has any failures is not very useful. However, there are two factors to take in to account in this context: a) frequent failures can lead users to stop running a test suite leading to further regressions b) long-term repeated failures can desensitize users leading them to ignore *new* failures facilitating further regressions Thus it's really a question of what context you're talking about running the test suite in. For purposes of Jenkins we want full visibility in to what is passing and what is failing and how long this has been going on for. Cheers. -K _______________________________________________ [email protected] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "[email protected]"
