On 12/05/2013 07:22 PM, Ademar de Souza Reis Jr. wrote:
On Wed, Dec 04, 2013 at 02:54:07PM -0500, Chris Evich wrote:
All,
Currently virt-tests attempts to detect when the current virtualization
environment can be recycled for use by the next test. Measurements show
this optimization saves a significant amount of testing time. However,
I believe the practical cost is significant additional maintenance
burdens, and (perhaps worse) greater-than-zero questionable test results.
On the maintenance-front, environment-cleanliness detection complexity
increases in proportion to additional hardware (and configuration)
support for both the harness and tests. This leads to the harness
requiring a lot of "magic" (complicated and distributed logic) code to
support cleanliness detection. I'm sure most seasoned developers here
have encountered failures in this area on more than a few occasions, and
have been exposed to pressure for complicated or messy fixes and
workarounds.
On the results-front, for all except the most simple tests, using the
default/automatic environment, PASS/FAIL trustworthiness is tied
directly to:
* Trust that the harness has managed expectations precisely for an
unknown number of proceeding tests.
* Assumption that the harness usually does the right thing over the
long-term. Otherwise tests can force environment reset/re-build when it
is critical.
After a lengthy discussion with lmr on the topic, we are questioning the
practical benefits of the time-savings versus the maintenance cost and
importance of long-term result-trust and reproduce-ability.
I believe we can significantly increase result-trust and reduce
maintenance burden (including "magic" code), if the harness takes an
"environment is always dirty" stance. In other words, take steps to
rebuild a known "pristine" environment between every test and
remove/reduce most of the cleanliness detection. Placing more of the
setup burden on the tests, which are closer to the state-requirements
anyway.
However, we feel it's important to also get the community's input on
this topic. Are most of you already using a combination of
'--restore-image-between-tests' and 'kill_vm = yes' anyway? Or do you
see large benefits from the harness doing cleanliness detection despite
the costs? What is your opinion or feedback on this topic?
Hi there.
This is a very nice topic to discuss, I'm glad you brought it.
As a wise man once said, "(Early) optimization is the root of all
evils". I hold the opinion that the *default* should be
'--restore-image-between-tests' and 'kill_vm = yes'.
Ok, I have more code and more data to back up the discussion. I have:
1) Fixed bugs that were making virt-test to behave erratically upon the
safe options
2) Updated JeOS, to an image with all the Fedora 19 updates to date
I must say this made the default run with all the safe options to be
*very* reasonable:
Today's defaults:
[lmr@thinkpad-t420s virt-test.git]$ ./run -t qemu
SETUP: PASS (1.70 s)
DATA DIR: /home/lmr/virt_test
DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2013-12-11-15.48.25/debug.log
TESTS: 10
(1/10) type_specific.migrate.default.tcp: PASS (38.26 s)
(2/10) type_specific.migrate.default.unix: PASS (28.27 s)
(3/10) type_specific.migrate.default.exec.default_exec: PASS (28.27 s)
(4/10) type_specific.migrate.default.exec.gzip_exec: PASS (30.21 s)
(5/10) type_specific.migrate.default.fd: PASS (28.07 s)
(6/10) type_specific.migrate.with_set_speed.tcp: PASS (24.39 s)
(7/10) type_specific.migrate.with_set_speed.unix: PASS (24.18 s)
(8/10) type_specific.migrate.with_set_speed.exec.default_exec: PASS (24.27 s)
(9/10) type_specific.migrate.with_set_speed.exec.gzip_exec: PASS (28.20 s)
(10/10) type_specific.migrate.with_set_speed.fd: PASS (24.32 s)
TOTAL TIME: 279.15 s (04:39)
TESTS PASSED: 10
TESTS FAILED: 0
SUCCESS RATE: 100.00 %
Safe options:
[lmr@thinkpad-t420s virt-test.git]$ ./run -t qemu --restart-vm --restore-image
--restore-image-between-tests
Running setup. Please wait...
SETUP: PASS (16.73 s)
DATA DIR: /home/lmr/virt_test
DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2013-12-11-15.55.43/debug.log
TESTS: 10
(1/10) type_specific.migrate.default.tcp: PASS (42.05 s)
(2/10) type_specific.migrate.default.unix: PASS (41.07 s)
(3/10) type_specific.migrate.default.exec.default_exec: PASS (40.48 s)
(4/10) type_specific.migrate.default.exec.gzip_exec: PASS (42.92 s)
(5/10) type_specific.migrate.default.fd: PASS (39.93 s)
(6/10) type_specific.migrate.with_set_speed.tcp: PASS (36.98 s)
(7/10) type_specific.migrate.with_set_speed.unix: PASS (37.12 s)
(8/10) type_specific.migrate.with_set_speed.exec.default_exec: PASS (37.57 s)
(9/10) type_specific.migrate.with_set_speed.exec.gzip_exec: PASS (43.40 s)
(10/10) type_specific.migrate.with_set_speed.fd: PASS (36.92 s)
TOTAL TIME: 399.37 s (06:39)
TESTS PASSED: 10
TESTS FAILED: 0
SUCCESS RATE: 100.00 %
Now 300s vs 400s in a more consistent way [1] makes this a very
interesting path to go.
Now, with these bugs fixed, we can go further and see how it looks like
in the test grid. Consider me excited with the prospect to change to
what I'll call to 'safe' mode by default!
[1] The previous JeOS caused some hangs that could drag the testing to
almost 600s.
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
