On Thu, Apr 19, 2018 at 09:29:55 +0200, Andrea Bolognani wrote: > On Wed, 2018-04-18 at 17:22 +0200, Peter Krempa wrote: > > On Wed, Apr 18, 2018 at 14:43:55 +0200, Andrea Bolognani wrote: > > > I think using real-life capabilities instead of the syntetic data > > > we use now is a step in the right direction. > > > > > > I'm still a bit uneasy about the "moving target" factor, though > > > I've spent some time trying to come up with possible failure > > > scenarios without much success... > > > > There are few churn-scenarios, some avoidable, some not so much: > > > > - When adding something which is added for a lot of configurations or all > > of them (e.g. recent adding of the new seccomp code) the change will > > generate a lot of changes. > > > > - Machine type. We can't use any of the alias machine types in the XML > > files as they will change when new capabilities with new machine types > > are added. This can easily be avoided by using an explicit machine > > type. > > > > While the above may induce some churn in some scenarios, I think that > > the benefit of at least seeing that something changed in places where > > you did not expect any change may outweigh it. > > I'm not really worried about churn, but about situations where > changes to libvirt would make it behave wrongly with old versions > of QEMU but correctly with newer ones, possibly with respect to > features that are not directly touched by the changes, in ways > unforeseen by the developer.
Well, that is actually exactly why we need to check against the latest capability set. If we have the locked-in feature bit set for testing, it will catch regressions which happen with that set of capabilities. Any regressing behaviour which would happen only with a newly added feature will not be caught, and those are the very sneaky ones. > Right now that's easy to spot because everything is locked to some > well-defined set of capabilities, but if we started using moving Well, only half of the scenarios is covered, as said above. > targets for most tests we'd decrease coverage of older QEMU in > favor of newer QEMU over time. I don't think this will happen if we fork the test for a certain version in cases when the output would change. If the output will not change we will not have to do anything. This will prevent having a lot of test files which don't do anything. > > > I think ideally with each new feature the author would introduce > > > three tests: a negative one locked to a QEMU version that doesn't > > > have the required bits, a positive one locked to a QEMU version > > > that has them, and a positive one against the latest QEMU version. > > > > I'm considering whether it's necessary to have the one locked to the > > older-but-supporting version of qemu. > > > > My idea was that if somebody were to add test which would change > > anything, the test can be forked prior to that. But it seems that it may > > be a better idea to lock-in the changes right away. > > I believe so. This would also make it so our test coverage always > increases over time. > > Now, a crazy idea: what if the test author would define a baseline > version QEMU where the test passes, and the test suite would > automatically test all versions between that one and the latest? It certainly is techincally possible as we could e.g. do a range of versions for the macro rather than single version. I'm only worried that it's slightly too expensive. The good thing is, that when we'll fork the tests rigorously on any change we should get coverage for most code paths. The unfortunate part is that the qemuxml2argvtest has now 800 XML files for testing (some are invalid though). If we introduce cartesian testing with all capability dumps, it will expand the test case count 12 fold which will tend to increase over time. (Thankfully Jano deleted a lot of old crap though) > That would mean a lot of churn whenever we add new capabilities > data, but it should be feasible to detect output files that do not > change and turn them into symlinks to reduce diffs and save disk > space. Actually if a test is stable (which requires it to be as minimal as possible) in a range of versions, adding new capabilities should be for free. We can test all versions in a range against the same output file. It should only change when adding a new feature. This only requires forking the test file if a change modifies anything retroactively. > We would also need a way to run a test only in a range of versions, > for negative testing. Well, the same as for positive testing. If we have infrastructure to test a version range, then this should be possible as well. I'm not sure though whether it's worth it at all. > Too far fetched? :) Slightly. But certainly has some value. Unfortunately I can't put in much more time besides this initial idea, since I'm busy with the blockdev stuff.
signature.asc
Description: PGP signature
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
