On 12/09/2013 11:43 PM, Chris Evich wrote:
On 12/08/2013 10:46 PM, Yiqiao Pu wrote:
Another use of this policy is making it easier to compose the simply
cases together to finish a more complex test scenario. Very useful for
some long term testing. Like in longevity testing we need all tests use
the same environment and keep guest always alive. Actually keeping using
a kind of dirty env is more useful in test for finding bugs. The real
env from customer can not always be clean.
Ahh, interesting, so we have some tests which _must_ maintain
environment, always. Is that for the _entire_ test job, or just some
collection of tests within a larger test job?
I think it is for the entire test job. The cases maybe some very simple
ones like boot, reboot, migrate and so on. But all of them need the vm
alive all the time. Otherwise you can not call that a longevity test. :)
And this really help us to reuse the tests in some situation. Like if I
have a test case:
1. do ping-pong migrate
2. ping the guest after the migrate finished
Then I can just write done a cfg file to run migrate and ping in the
right order. If we can not let the guest alive, we need add some code to
migrate to achieve this.
So there will be less duplicate code, simple case scripts and lots of
cases can be combined to new test loops for all kinds of testing and matrix.
And for the env setup did in pre/post process, we can set up parameters
that can be checked also in pre/post process. But for the operates
inside test, I think it is eaiser to let the tests themselves to try to
restore the status that might influence other cases. It is not hard to
The challenge we face here is tests have no insight into the test-list,
so they cannot know what test is coming next. We could possibly make
this information available, however I think this would just be moving a
lot of complexity out of pre/post, and duplicating it many times in
tests instead. Either way, I think you are right about letting tests
decide what to do, rather then have harness try to be "smart".
However, if we take an 'always restore between tests' stance by default,
it's less code for tests AND less code for pre/post. For the longevity
testing specifically, that I imagine having a special-case
flag/param/option where pre/post simply do nothing (or very minimal
checking only), and the test can maintain the env in this case.
This can be configurable. But the very basic problem is how we handle it
in the env. As now the biggest problem is in vm object as some options
is mixed up with new and old data. So I think the key issue is when we
get a vm from pre case in env, we didn't treat the object as an old
object, we just use it. When we get the vm from env, we should qualify
it first, during this stage we should protect it from any kind of
update. And in this stage there should have no operates about the
current case setup(for both env and guest). So we can do some clean up
if the vm is not qualified with the old data, if not just keep it.
After this judgement the preprocess should really go to the current
case(env setup and guest setup) with the right vm object and parameters.
outside our guest and may need be used more than one case. And another
one is some special operate inside guests which should be recovered
immediately.
Sure, and for these special-cases I agree it would be better to handle
them inside the tests. If many tests need do same pre/post, they can
always share common utils.test code, or import code from other test
module. Combine this with 'do nothing' signal to harness, and I am
hopeful we can reduce harness complexity but not take away power from tests.
Are there other testing situations that would not be addressed by having
just harness 'always restore', and harness 'do nothing' modes?
Basically 'always restore' is for some operates that may break the guest
and env. (like some blk tests may break image, boundary tests may set up
some unusual parameters), and 'do nothing' is mostly for some env setup
like hugepage, image backend, sriov and so on.
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
