On Sat, Oct 25, 2014 at 2:46 PM, Garrett Cooper <yaneurab...@gmail.com> wrote:
> On Oct 25, 2014, at 13:20, K. Macy <km...@freebsd.org> wrote:
>
>>>>> Alan also suggested against integrating the test suite as-is, because as 
>>>>> he said, "Remember, don't run these tests on a production system.  They 
>>>>> WILL cause panics and deadlocks, and they may cause data loss too.”
>>>>>
>>>>> Cheers,
>>>>> -Garrett
>>>>
>>>> Wait, we want to sweep those bugs under the rug?  What exactly is wrong 
>>>> with making a test harness that can very easily reproduce a known problem? 
>>>>  The chances are that anyone will dive into it once the bug is easily 
>>>> reproducible.
>>>
>>>        Sweeping bugs under the rug is not what I plan on doing; I’m marking 
>>> these as expected failures, as opposed to having them continually panic a 
>>> machine. Once a ZFS dev takes a look at the issue and resolves them, then 
>>> the ZFS dev can remove the “bail” calls I’m adding to the testcases.
>>
>> Yes, disabling tests that fail leads to an ineffectual test suite. A
>> test suite that never has any failures is not very useful. However,
>> there are two factors to take in to account in this context:
>> a) frequent failures can lead users to stop running a test suite
>> leading to further regressions
>> b) long-term repeated failures can desensitize users leading them to
>> ignore *new* failures facilitating further regressions
>>
>> Thus it's really a question of what context you're talking about
>> running the test suite in. For purposes of Jenkins we want full
>> visibility in to what is passing and what is failing and how long this
>> has been going on for.
>
> (seeing as how my other post isn’t in the -testing archives yet..)
>         Panicking a node (what the tests are doing before last night) and 
> exiting with a non-zero exit code (what I’m making them do with the bail outs 
> in tools/regression/zfs) are both considered test failures. The difference 
> being that I can safely run all of the tests on a production or a test 
> machine without having to panic/reboot the box and I get greater coverage in 
> one fell swoop. If a developer wants they can always delete the lines that 
> bail out of the tests to get the desired panic.

I don't think there is a RIGHT answer. I just ask that it be noisy
about the choice that's being made. In other words, it should be very
explicit that it's running a "safe" subset of the tests because
FreeBSD's ZFS can't actually pass all of them, and that there be a
flag to enable those of us who want to see the failure to see it
without modifying any scripts or config files. A test framework that
requires me to muck with settings to run failing tests isn't living up
to its full potential.


-K
_______________________________________________
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Reply via email to