[vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. -- Thanks and best regards! Zhou Zheng Sheng / 周征晟 E-mail: zhshz...@linux.vnet.ibm.com Telephone: 86-10-82454397 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
On 08/08/2012 10:37 PM, Zhou Zheng Sheng wrote: Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. We found this when we started running 5 of them at once. I have since limited the job to one per host. To stop the failures. I though I had resubmitted all the jobs that failed because of that. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. This information is useful to know. I wonder if we could just remove the stress test from the patch test and just leave it on the master branch tests. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
On 08/09/2012 05:46 AM, Robert Middleswarth wrote: On 08/08/2012 10:37 PM, Zhou Zheng Sheng wrote: Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. We found this when we started running 5 of them at once. I have since limited the job to one per host. To stop the failures. I though I had resubmitted all the jobs that failed because of that. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. This information is useful to know. I wonder if we could just remove the stress test from the patch test and just leave it on the master branch tests. or you could do a separate job running only this test separate from all the other tests, and limit this job to one per host. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
