Re: kvm-autotest: The automation plans?
sudhir kumar 写道: 2009/5/14 sudhir kumar smalik...@gmail.com: On Thu, May 14, 2009 at 12:22 PM, jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. Cool. I would like to have look on them. So how do you manage the background process/thread? (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Not much clue here. But I would like to keep the control file as simple as possible and as much independent of test scenarios as possible. kvm_tests.cfg should be the right file untill and unless it is impossible to do by using it. 2) Public bridge support: I've sent a patch(TAP network support in Did you send the patch on the mailing list? Can you please point me to it. In my knowledge -redir tcp *:* breaks with -net tap. If running an external dhcp/dns server is the solution then the patch should go in. Thanks in advance!! Thanks for pointing out this problem. The redirections indeed breaks the tap networking. TAP patch could be seen at http://marc.info/?l=kvmm=124221382604554w=2. The redirections also leads the problems of getting the ip address of a user mode nic. So maybe we could make the tap/user to be an exclusive option. kvm-autotest), this patch needs external DHCP server and requires nmap support. I don't know whether the method of original kvm_runtes_old(DHCP server of private bridge) is preferable. The old approach is better. All might not be able to run an external DHCP server for running the test. I do not see any issue with the old approach. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
Michael Goldish 写道: - sudhir kumar smalik...@gmail.com wrote: On Thu, May 14, 2009 at 12:22 PM, jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. Cool. I would like to have look on them. So how do you manage the background process/thread? Yes, we would try to sent it here as soon as possible. The background workload could be added through various methods. We could an simple algorithm as follows: run_migration2(): pid = run_autotest_background(test,params,env,dbench,control.60) Do ping-pong migration ... wait_autoteset_background(pid) run_autotest_background() would fork a subprocess to run function run_autotest() and catch its exception. wait_autotest_background(pid) would wait until the background benchmark complete and analyse the result through the return value of the subprocess. The child process could work well depends the fact that the ssh connection should alive during migration. I believe this could be also achieved through job.parallel() (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Not much clue here. But I would like to keep the control file as simple as possible and as much independent of test scenarios as possible. kvm_tests.cfg should be the right file untill and unless it is impossible to do by using it. 2) Public bridge support: I've sent a patch(TAP network support in kvm-autotest), this patch needs external DHCP server and requires nmap support. I don't know whether the method of original kvm_runtes_old(DHCP server of private bridge) is preferable. The old approach is better. All might not be able to run an external DHCP server for running the test. I do not see any issue with the old approach. We're taking more of a minimalist approach in kvm_runtest_2: the framework should handle only the things directly related to testing. Configuring and running a DHCP server is and should be beyond the scope of the KVM-Autotest framework. To emulate the old behavior, you can just start the DHCP server yourself locally. If you wish, maybe we can bundle example scripts with the framework that will do this for the user, but they should not be an integral part of the framework in my opinion. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
- jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We'll probably have to use file locks anyway when we work with TAP, but in VM.create(), not in get_free_port(), because we also want to prevent parallel qemu instances from choosing the same TAP device. I'm not sure how qemu handles this internally, and I'd rather be on the safe side. Do you release the file lock inside get_free_port or only after running qemu? We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Would it not suffice to just modify the configuration, instead of completely define it, inside the control file? This is possible using parse_string(). For example: cfg = kvm_config.config(kvm_tests.cfg) cfg.parse_string(only weekly) cfg.parse_string(only Fedora RHEL Windows) cfg.parse_string( variants: - 1: only ide - 2: Fedora: no rtl8139 ) list = cfg.get_list() (get_list() returns the test dictionaries.) The advantage here is that we can have a standard kvm_tests.cfg that we all agree on and only rather small environment-specific modifications are made in the control file. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
On Wed, May 13, 2009 at 11:30 PM, Michael Goldish mgold...@redhat.com wrote: - sudhir kumar smalik...@gmail.com wrote: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. Yes, we have plans to implement such functionality. It shouldn't be hard, but we need to give it some thought in order to implement it as elegantly as possible. I completely agree here. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? I currently have some experimental patches that allow running of several parallel queues of tests. But what exactly do you mean by Please post them. N parallel instances of a test? Do you mean N queues? Please provide an example so I can get a better idea. I wanted a parallelism in 2 degrees. Let me try with an example. The following test only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench is just one instance and will create one VM with given specifications and execute migrate and dbench. So I am thinking how can we trigger n similar tests execution in parallel. I feel job.parallel() is meant for that but is kvm_tests.cfg good enough to be used under such a scenario? However we have most of the stuff non static(as getting the free vnc port, etc) but still we have some variables which are static. For ex. vm name, migration port etc. So what are your thoughts on it. In this scenario my system will be having N VMs, all running the same set of testcases. On the other hand I was looking for something like this as well. only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench.dbench_instancesN.bonnie Thus all the tests will be executed in normal way except dbench. There should be running N instances of dbench and when over simply run bonnie and exit. I hope my demand to kvm-autotest is not too much but for an effective and rigorous testing of kvm such a framework is necessary. I am bit new to autotest framework and have very little knowledge of the server side. I will start spending some time on looking at the available features. Hope I was clear this time. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
On Thu, May 14, 2009 at 12:22 PM, jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. Cool. I would like to have look on them. So how do you manage the background process/thread? (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Not much clue here. But I would like to keep the control file as simple as possible and as much independent of test scenarios as possible. kvm_tests.cfg should be the right file untill and unless it is impossible to do by using it. 2) Public bridge support: I've sent a patch(TAP network support in kvm-autotest), this patch needs external DHCP server and requires nmap support. I don't know whether the method of original kvm_runtes_old(DHCP server of private bridge) is preferable. The old approach is better. All might not be able to run an external DHCP server for running the test. I do not see any issue with the old approach. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
- sudhir kumar smalik...@gmail.com wrote: On Thu, May 14, 2009 at 12:22 PM, jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. Cool. I would like to have look on them. So how do you manage the background process/thread? (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Not much clue here. But I would like to keep the control file as simple as possible and as much independent of test scenarios as possible. kvm_tests.cfg should be the right file untill and unless it is impossible to do by using it. 2) Public bridge support: I've sent a patch(TAP network support in kvm-autotest), this patch needs external DHCP server and requires nmap support. I don't know whether the method of original kvm_runtes_old(DHCP server of private bridge) is preferable. The old approach is better. All might not be able to run an external DHCP server for running the test. I do not see any issue with the old approach. We're taking more of a minimalist approach in kvm_runtest_2: the framework should handle only the things directly related to testing. Configuring and running a DHCP server is and should be beyond the scope of the KVM-Autotest framework. To emulate the old behavior, you can just start the DHCP server yourself locally. If you wish, maybe we can bundle example scripts with the framework that will do this for the user, but they should not be an integral part of the framework in my opinion. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
- sudhir kumar smalik...@gmail.com wrote: On Wed, May 13, 2009 at 11:30 PM, Michael Goldish mgold...@redhat.com wrote: - sudhir kumar smalik...@gmail.com wrote: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. Yes, we have plans to implement such functionality. It shouldn't be hard, but we need to give it some thought in order to implement it as elegantly as possible. I completely agree here. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? I currently have some experimental patches that allow running of several parallel queues of tests. But what exactly do you mean by Please post them. N parallel instances of a test? Do you mean N queues? Please provide an example so I can get a better idea. I wanted a parallelism in 2 degrees. Let me try with an example. The following test only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench is just one instance and will create one VM with given specifications and execute migrate and dbench. So I am thinking how can we trigger n similar tests execution in parallel. I feel job.parallel() is meant for that but is kvm_tests.cfg good enough to be used under such a scenario? However we have most of the stuff non static(as getting the free vnc port, etc) but still we have some variables which are static. For ex. vm name, migration port etc. So what are your thoughts on it. I think generally kvm_tests.cfg is flexible enough, and can easily be modified to define whatever you like. Note, however, that the config file parser module is only responsible for producing a list of dictionaries which define the tests to run. It doesn't care much about parallelism -- this is up to the control file and the rest of the framework. If you're not familiar with the format of config files, please refer to http://www.linux-kvm.org/page/KVM-Autotest/Test_Config_File and http://www.linux-kvm.org/page/KVM-Autotest/Parameters In this scenario my system will be having N VMs, all running the same set of testcases. I thought you said one VM running migrate and dbench in parallel. I'm not sure I follow. On the other hand I was looking for something like this as well. only raw.*ide.*default.*smp2.*RHEL5.3.i386.*migrate.dbench.dbench_instancesN.bonnie Thus all the tests will be executed in normal way except dbench. There should be running N instances of dbench and when over simply run bonnie and exit. This seems like two tests to me: dbench with dbench (several instances), and then another unrelated bonnie test. Also note that the variants you select with 'only' must be defined before they can be selected. Look at the examples in the wiki as well as real config files. I hope my demand to kvm-autotest is not too much but for an effective and rigorous testing of kvm such a framework is necessary. I am bit new to autotest framework and have very little knowledge of the server side. I will start spending some time on looking at the available features. Hope I was clear this time. Regarding parallelism: Generally two types can be implemented. 1. Several independent test execution queues: in this case there are several queues that don't interfere with each other. Each queue works with its own VMs. This is useful for saving time by running tests in parallel on capable hosts. This can be implemented using job.parallel() and is already running in TLV. I will try to post the patches soon. This can probably also be implemented from the server, if it can treat a single physical host as if it were several, thus running several independent copies of the Autotest client on it. 2. Several tests on a single VM, which is what you were referring to, if I understood correctly: in this case several threads work with the same VMs and abuse them in parallel -- one thread can run dbench while the other runs migration on the same VM. This is possible using threads, and the syntax in the config file can be something like 'types = dbench migration' instead of what we currently use -- 'type = dbench'. However, we have to think whether we really just want to run tests in parallel. In the migration-dbench case, for example, we'd like to make sure dbench starts running before we migrate. So maybe it's wiser to just run some load inside the migration test, instead of the dbench test. We should carefully consider all options. Thanks, Michael -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a
Re: kvm-autotest: The automation plans?
Michael Goldish 写道: - jason wang jasow...@redhat.com wrote: sudhir kumar 写道: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. We have some patches of ping pong migration and workload adding. The migration is based on public bridge and workload adding is based on running benchmark in the background of guest. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. The parallelized instances could be easily achieved through job.parallel() of autotest framework, and that is what we have used in our tests. We have make some helper routines such as get_free_port to be reentrant through file lock. We'll probably have to use file locks anyway when we work with TAP, but in VM.create(), not in get_free_port(), because we also want to prevent parallel qemu instances from choosing the same TAP device. I'm not sure how qemu handles this internally, and I'd rather be on the safe side. Do you release the file lock inside get_free_port or only after running qemu? We record the port usage and release the file lock inside get_free_port(). I agree with you that it's better to get/release the file lock in VM.create() because it is easier and it also eliminates the effort of doing lock in every helper function. For the TAP device, maybe we could give each TAP device used by qemu-kvm an random generated ifname to prevent qemu-kvm from choosing the same TAP devices. This method works well in our test. We've implemented following test cases: timedrift(already sent here), savevm/loadvm, suspend/resume, jumboframe, migration between two machines and others. We will sent it here for review in the following weeks. There are some other things could be improved: 1) Current kvm_test.cfg.sample/kvm_test.cfg is transparent to autotest server UI. This would make it hard to configure the tests in the server side. During our test, we have merged it into control and make it could be configured by editing control file function of autotest server side web UI. Would it not suffice to just modify the configuration, instead of completely define it, inside the control file? This is possible using parse_string(). For example: cfg = kvm_config.config(kvm_tests.cfg) cfg.parse_string(only weekly) cfg.parse_string(only Fedora RHEL Windows) cfg.parse_string( variants: - 1: only ide - 2: Fedora: no rtl8139 ) list = cfg.get_list() (get_list() returns the test dictionaries.) The advantage here is that we can have a standard kvm_tests.cfg that we all agree on and only rather small environment-specific modifications are made in the control file. Thanks, this way makes the things easier. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
kvm-autotest: The automation plans?
Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? Please provide your thoughts on the above features. -- Sudhir Kumar -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
- sudhir kumar smalik...@gmail.com wrote: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: (1). we have standalone migration only. Is there any plans of enhancing kvm-autotest so that we can trigger migration while a workload is running? Something like this: Start a workload(may be n instances of it). let the test execute for some time. Trigger migration. Log into the target. Check if the migration is succesful Check if the test results are consistent. Yes, we have plans to implement such functionality. It shouldn't be hard, but we need to give it some thought in order to implement it as elegantly as possible. (2). How can we run N parallel instances of a test? Will the current configuration be easily able to support it? I currently have some experimental patches that allow running of several parallel queues of tests. But what exactly do you mean by N parallel instances of a test? Do you mean N queues? Please provide an example so I can get a better idea. Thanks, Michael -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kvm-autotest: The automation plans?
On Wed, 2009-05-13 at 23:21 +0530, sudhir kumar wrote: Hi Uri/Lucas, Do you have any plans for enhancing kvm-autotest? I was looking mainly on the following 2 aspects: Hi Sudhir, about the two questions you've made, Michael has answered them a lot better than I possibly could. So please keep in touch and send your ideas so we can consider implementing them on our tests! Thank you very much, -- Lucas Meneghel Rodrigues Software Engineer (QE) Red Hat - Emerging Technologies -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
