>>> Alan also suggested against integrating the test suite as-is, because as he 
>>> said, "Remember, don't run these tests on a production system.  They WILL 
>>> cause panics and deadlocks, and they may cause data loss too.”
>>>
>>> Cheers,
>>> -Garrett
>>
>> Wait, we want to sweep those bugs under the rug?  What exactly is wrong with 
>> making a test harness that can very easily reproduce a known problem?  The 
>> chances are that anyone will dive into it once the bug is easily 
>> reproducible.
>
>         Sweeping bugs under the rug is not what I plan on doing; I’m marking 
> these as expected failures, as opposed to having them continually panic a 
> machine. Once a ZFS dev takes a look at the issue and resolves them, then the 
> ZFS dev can remove the “bail” calls I’m adding to the testcases.
> Cheers,
> -Garrett

Yes, disabling tests that fail leads to an ineffectual test suite. A
test suite that never has any failures is not very useful. However,
there are two factors to take in to account in this context:
a) frequent failures can lead users to stop running a test suite
leading to further regressions
b) long-term repeated failures can desensitize users leading them to
ignore *new* failures facilitating further regressions

Thus it's really a question of what context you're talking about
running the test suite in. For purposes of Jenkins we want full
visibility in to what is passing and what is failing and how long this
has been going on for.

Cheers.

-K
_______________________________________________
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Reply via email to