On Sat, Oct 25, 2014 at 1:20 PM, K. Macy <[email protected]> wrote: >>>> Alan also suggested against integrating the test suite as-is, because as >>>> he said, "Remember, don't run these tests on a production system. They >>>> WILL cause panics and deadlocks, and they may cause data loss too.” >>>> >>>> Cheers, >>>> -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong >>> with making a test harness that can very easily reproduce a known problem? >>> The chances are that anyone will dive into it once the bug is easily >>> reproducible. >> >> Sweeping bugs under the rug is not what I plan on doing; I’m marking >> these as expected failures, as opposed to having them continually panic a >> machine. Once a ZFS dev takes a look at the issue and resolves them, then >> the ZFS dev can remove the “bail” calls I’m adding to the testcases. >> Cheers, >> -Garrett > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for.
Agreed. We're talking about doing an OpenZFS bug-tracker and maybe we could have tests post there automatically, once the bug is opened. I am going to put this on the board at MeetBSD Ca. next week, so we can have a discussion about it in real-time with some of the people present. Cheers, -matt _______________________________________________ [email protected] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "[email protected]"
