On 12/05/2013 03:54 AM, Chris Evich wrote:
All,
Currently virt-tests attempts to detect when the current virtualization
environment can be recycled for use by the next test. Measurements show
this optimization saves a significant amount of testing time. However,
I believe the practical cost is significant additional maintenance
burdens, and (perhaps worse) greater-than-zero questionable test results.
On the maintenance-front, environment-cleanliness detection complexity
increases in proportion to additional hardware (and configuration)
support for both the harness and tests. This leads to the harness
requiring a lot of "magic" (complicated and distributed logic) code to
support cleanliness detection. I'm sure most seasoned developers here
have encountered failures in this area on more than a few occasions, and
have been exposed to pressure for complicated or messy fixes and
workarounds.
On the results-front, for all except the most simple tests, using the
default/automatic environment, PASS/FAIL trustworthiness is tied
directly to:
* Trust that the harness has managed expectations precisely for an
unknown number of proceeding tests.
* Assumption that the harness usually does the right thing over the
long-term. Otherwise tests can force environment reset/re-build when it
is critical.
After a lengthy discussion with lmr on the topic, we are questioning the
practical benefits of the time-savings versus the maintenance cost and
importance of long-term result-trust and reproduce-ability.
I believe we can significantly increase result-trust and reduce
maintenance burden (including "magic" code), if the harness takes an
"environment is always dirty" stance. In other words, take steps to
rebuild a known "pristine" environment between every test and
remove/reduce most of the cleanliness detection. Placing more of the
setup burden on the tests, which are closer to the state-requirements
anyway.
However, we feel it's important to also get the community's input on
this topic. Are most of you already using a combination of
'--restore-image-between-tests' and 'kill_vm = yes' anyway? Or do you
see large benefits from the harness doing cleanliness detection despite
the costs? What is your opinion or feedback on this topic?
Another use of this policy is making it easier to compose the simply
cases together to finish a more complex test scenario. Very useful for
some long term testing. Like in longevity testing we need all tests use
the same environment and keep guest always alive. Actually keeping using
a kind of dirty env is more useful in test for finding bugs. The real
env from customer can not always be clean.
And for the env setup did in pre/post process, we can set up parameters
that can be checked also in pre/post process. But for the operates
inside test, I think it is eaiser to let the tests themselves to try to
restore the status that might influence other cases. It is not hard to
handle the iscsi setup things outside tests, but hard to know if pre
case set up the migrate_speed to a very low value, what value(the
default value in system) should I reset it to inside the case. So
basically we can separate this to two part, one is some env setup
outside our guest and may need be used more than one case. And another
one is some special operate inside guests which should be recovered
immediately.
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
