On Tue, 2018-02-27 at 09:59 -0500, Laine Stump wrote:
> > So should I just drop this one, or is it still somewhat valuable
> > to have any sort of test suite coverage for PCI controller options?
> I was thinking that having this test is better than not having this
> test. But then I thought about what would happen  if there was a
> regression in just a single one of these validations - the negative test
> would still "succeed" (i.e. "succeed in detecting a failure") because it
> would hit the next disallowed attribute. As a matter of fact, the test
> would continue to "succeed" until there was a regression in the
> validation of every single attribute, so that no error would be
> triggered. So having a negative test that has multiple examples of
> failures actually gives us a false sense of security - we believe that
> it's verifying we're catching incorrect config, but it won't actually
> notify us until *all* of the bad config is missed by validation.
> So, I think each negative test should have exactly one piece of
> incorrect data. That necessarily means that it's only testing for proper
> validation of a single aspect of a single attribute. But making this
> generally useful with the current test apparatus would mean a huge
> explosion in the number of test files, and I don't think that's
> practical. But if we're only testing for one out of a thousand
> validations, there's really not much point in it.

Okay, I'll drop it and respin then.

Andrea Bolognani / Red Hat / Virtualization

libvir-list mailing list

Reply via email to