All, Currently virt-tests attempts to detect when the current virtualization environment can be recycled for use by the next test. Measurements show this optimization saves a significant amount of testing time. However, I believe the practical cost is significant additional maintenance burdens, and (perhaps worse) greater-than-zero questionable test results.
On the maintenance-front, environment-cleanliness detection complexity increases in proportion to additional hardware (and configuration) support for both the harness and tests. This leads to the harness requiring a lot of "magic" (complicated and distributed logic) code to support cleanliness detection. I'm sure most seasoned developers here have encountered failures in this area on more than a few occasions, and have been exposed to pressure for complicated or messy fixes and workarounds. On the results-front, for all except the most simple tests, using the default/automatic environment, PASS/FAIL trustworthiness is tied directly to: * Trust that the harness has managed expectations precisely for an unknown number of proceeding tests. * Assumption that the harness usually does the right thing over the long-term. Otherwise tests can force environment reset/re-build when it is critical. After a lengthy discussion with lmr on the topic, we are questioning the practical benefits of the time-savings versus the maintenance cost and importance of long-term result-trust and reproduce-ability. I believe we can significantly increase result-trust and reduce maintenance burden (including "magic" code), if the harness takes an "environment is always dirty" stance. In other words, take steps to rebuild a known "pristine" environment between every test and remove/reduce most of the cleanliness detection. Placing more of the setup burden on the tests, which are closer to the state-requirements anyway. However, we feel it's important to also get the community's input on this topic. Are most of you already using a combination of '--restore-image-between-tests' and 'kill_vm = yes' anyway? Or do you see large benefits from the harness doing cleanliness detection despite the costs? What is your opinion or feedback on this topic? -- Chris Evich, RHCA, RHCE, RHCDS, RHCSS Quality Assurance Engineer e-mail: cevich + `@' + redhat.com o: 1-888-RED-HAT1 x44214 _______________________________________________ Virt-test-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/virt-test-devel
