[vdsm] Jenkins and Gerrit.
I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Jenkins and Gerrit.
On Wed, Aug 08, 2012 at 07:47:02AM -0400, Robert Middleswarth wrote: I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. Thanks, Robert, for working on this. It is highly important for me to know that something is going to break the build before taking it in. However, would it be possible to have a repository where we can review the code of the robot? I think it is important for the robot to be less noisy, and particularly, never give V+1. This task is reserved to humans that actually know what the patch should be doing. Also, I am not at all sure that the robot is limitting itself to be running code of trustworthy authors. Regards, Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Jenkins and Gerrit.
On Wed, Aug 08, 2012 at 03:48:13PM +0300, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 07:47:02AM -0400, Robert Middleswarth wrote: I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. Thanks, Robert, for working on this. It is highly important for me to know that something is going to break the build before taking it in. However, would it be possible to have a repository where we can review the code of the robot? It's Gerrit Trigger[1] and the code is on github[2]. I think it is important for the robot to be less noisy, and particularly, never give V+1. This task is reserved to humans that actually know what the patch should be doing. The V+1 has been fixed. Will give 0 when they pass, -1 when they fail. Also, I am not at all sure that the robot is limitting itself to be running code of trustworthy authors. Eyal added a feature request for this[3]. This was the result of a discussion on the infra mailing list[4]. [1]: https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger [2]: https://github.com/jenkinsci/gerrit-trigger-plugin [3]: https://issues.jenkins-ci.org/browse/JENKINS-14655 [4]: http://lists.ovirt.org/pipermail/infra/2012-August/000759.html ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Jenkins and Gerrit.
On 08/08/2012 08:48 AM, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 07:47:02AM -0400, Robert Middleswarth wrote: I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. Thanks, Robert, for working on this. It is highly important for me to know that something is going to break the build before taking it in. However, would it be possible to have a repository where we can review the code of the robot? It is the Jenkins Gerrit Tigger bot direct from jenkins-ci.org that is running the same unit jenkins has been running just against new patches. I think it is important for the robot to be less noisy, and particularly, never give V+1. Already adjusted new runs will give -1 for fail but 0 for success going forward. There were about 400 jobs/patches it gave +1 to if needed those can be re-run please let me know is this is required. This task is reserved to humans that actually know what the patch should be doing. Also, I am not at all sure that the robot is limitting itself to be running code of trustworthy authors. That is correct the reason there were some fails last night was my attempt to add in the limit and I broke things for a few min. I have already tested all the logic and should be in place by the end of today. Regards, Dan. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Jenkins and Gerrit.
On Wed, Aug 08, 2012 at 02:55:17PM +0200, Ewoud Kohl van Wijngaarden wrote: On Wed, Aug 08, 2012 at 03:48:13PM +0300, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 07:47:02AM -0400, Robert Middleswarth wrote: I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. Thanks, Robert, for working on this. It is highly important for me to know that something is going to break the build before taking it in. However, would it be possible to have a repository where we can review the code of the robot? It's Gerrit Trigger[1] and the code is on github[2]. I think it is important for the robot to be less noisy, and particularly, never give V+1. This task is reserved to humans that actually know what the patch should be doing. The V+1 has been fixed. Will give 0 when they pass, -1 when they fail. Also, I am not at all sure that the robot is limitting itself to be running code of trustworthy authors. Eyal added a feature request for this[3]. This was the result of a discussion on the infra mailing list[4]. As much as I like (and need) this per-commit verification, I think we should not deploy it before the feature is implemented. BTW, Federico suggested to initiate the test only on request (when oVirt Jenkins CI Server is added as reviewer). This would allow a more silent start for CI. Thanks, Dan. [1]: https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger [2]: https://github.com/jenkinsci/gerrit-trigger-plugin [3]: https://issues.jenkins-ci.org/browse/JENKINS-14655 [4]: http://lists.ovirt.org/pipermail/infra/2012-August/000759.html ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] minutes: today's call
* Itamar Heim ih...@redhat.com [2012-08-01 08:27]: On 08/01/2012 04:20 PM, Ryan Harper wrote: * Itamar Heim ih...@redhat.com [2012-07-25 08:50]: On 07/25/2012 04:36 PM, Ryan Harper wrote: * Eyal Edri ee...@redhat.com [2012-07-18 11:59]: I'm in favor on doing it next monday. (not too healthy to do upgrades before the weekend...). We'll need to send email to rhev-devel + qe for an estimated downtime of 1 hour for gerrit.eng.lab.tlv.redhat.com. Any update? I've sent an update earlier today to infra that i'm planning to upgrade gerrit to 2.4.2 coming sunday. after a few days of a clean upgrade, I'll do the patched version of it. How'd the upgrade go? any ETA for when the patched version will show up? too early to say - gerrit already required a restart today after hanging. Gal already created a patched version for me to deploy, but i want to see the gerrit behavior for a few more days before i apply it. (btw, i think gerrit 2.5 has some improvements over the original patches (like max size, etc.), but we'll wait with them for 2.5 i guess. How's gerrit behaving? Ready for the patched version? Thanks! Ryan Ryan Eyal. - Original Message - From: Itamar Heim ih...@redhat.com To: Ryan Harper ry...@us.ibm.com Cc: Dan Kenigsberg dan...@redhat.com, Adam Litke a...@us.ibm.com, Anthony Liguori aligu...@us.ibm.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org, Eyal Edri ee...@redhat.com, Attila Darazs adar...@redhat.com, Moran Goldboim mgold...@redhat.com Sent: Wednesday, July 18, 2012 7:48:01 PM Subject: Re: [vdsm] minutes: today's call On 07/18/2012 06:24 PM, Ryan Harper wrote: * Itamar Heim ih...@redhat.com [2012-05-16 10:26]: On 05/16/2012 06:11 PM, Dan Kenigsberg wrote: On Wed, May 16, 2012 at 09:43:57AM -0500, Ryan Harper wrote: * Dan Kenigsbergdan...@redhat.com [2012-05-07 05:42]: On Mon, Apr 23, 2012 at 05:52:13PM +0300, Dan Kenigsberg wrote: On Mon, Apr 23, 2012 at 07:34:14AM -0500, Adam Litke wrote: On Mon, Apr 23, 2012 at 04:17:18AM -0400, Ayal Baron wrote: Hi all, I would like to discuss the following on today's call: 1. Gerrit vs. mailing list Gerrit is an inhibiter for some contributors. One approach to solve this improve gerrit: - Gerrit should send the patch when it notified of a change. This may attract more reviewers. I'm happy to inform that Gal has sent a patch for this to upstream gerrit: https://gerrit-review.googlesource.com/#/c/34861/ Add unified diff to newchange mail template. Any eta on getting the gerrit notifications of changes to include the full patch in the email? You can +1 them in googlesource, maybe it helps ;-) indeed, please help push them upstream... https://gerrit-review.googlesource.com/#/c/34861/ https://gerrit-review.googlesource.com/#/c/34862/ I'm definitely happy to see notification on new posts and changes; helps me see what new activity is happening, but I'd really enjoy seeing the the patch series attached (via threading) as well. Itamar, I know Red Hat hates to do it, but can we take these patches to ovirt's gerrit before they are accepted upstream? yes we can. after i'll upgrade it to 2.3, which i prefer to do after i return from PTO in case there are some issues with the upgrade. so ETA end of the month for clean upgrade to 2.3, then after we see all is ok, let's say another week to upgrade to the version with these patches. It's about 6 weeks later. Dan's been asking for more review in vdsm; I know that I'd be able to review quite a bit more if I can get patches via email instead of the webui. Can we get this enabled? eyal/attila - any ETA on testing gerrit 2.4 (.2 by now) so we can upgrade to it? -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx ry...@us.ibm.com ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Jenkins and Gerrit.
On 08/08/2012 03:07 PM, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 09:58:17AM -0400, Robert Middleswarth wrote: On 08/08/2012 09:50 AM, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 02:55:17PM +0200, Ewoud Kohl van Wijngaarden wrote: On Wed, Aug 08, 2012 at 03:48:13PM +0300, Dan Kenigsberg wrote: On Wed, Aug 08, 2012 at 07:47:02AM -0400, Robert Middleswarth wrote: I have setup patch review on Jenkins.info for newly submitted patches and it seems to be working pretty well over all but last night well tweaking the process I broken it for a few min but that was long enough that about 50 jobs were marked -1 I will be fixing that today by rerunning the jobs. I am sorry if one of your patches was dinged and it should be fixed by this time tomorrow. Thanks, Robert, for working on this. It is highly important for me to know that something is going to break the build before taking it in. However, would it be possible to have a repository where we can review the code of the robot? It's Gerrit Trigger[1] and the code is on github[2]. I think it is important for the robot to be less noisy, and particularly, never give V+1. This task is reserved to humans that actually know what the patch should be doing. The V+1 has been fixed. Will give 0 when they pass, -1 when they fail. Also, I am not at all sure that the robot is limitting itself to be running code of trustworthy authors. Eyal added a feature request for this[3]. This was the result of a discussion on the infra mailing list[4]. As much as I like (and need) this per-commit verification, I think we should not deploy it before the feature is implemented. BTW, Federico suggested to initiate the test only on request (when oVirt Jenkins CI Server is added as reviewer). This would allow a more silent start for CI. Thanks, Dan. I already wrote a little bash code to do this outside the plug-in. It will be in place by the end of the day. This kind of script is exactly the thing I'd like to be peer-reviewed before applied en mass to gerrit changes. Particularly due to the security implications. Regards, Dan. If you are talking about the jenkins app that updates Gerrit that is has been in use on ovirt-node-devel for some time. As for the whitelist script that is like 4 lines. git log --pretty=%ce -n 1 $WORKSPACE/current_author.txt grep -f $WORKSPACE/current_author.txt $WORKSPACE/jenkins-whitelist.txt RETVAL=$? [ $RETVAL -ne 0 ] curl -u jenkins_bot:xx $BUILD_URL/stop; It is simple and the files are generated outside of the repo so it should be safe. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. -- Thanks and best regards! Zhou Zheng Sheng / 周征晟 E-mail: zhshz...@linux.vnet.ibm.com Telephone: 86-10-82454397 ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
On 08/08/2012 10:37 PM, Zhou Zheng Sheng wrote: Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. We found this when we started running 5 of them at once. I have since limited the job to one per host. To stop the failures. I though I had resubmitted all the jobs that failed because of that. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. This information is useful to know. I wonder if we could just remove the stress test from the patch test and just leave it on the master branch tests. -- Thanks Robert Middleswarth @rmiddle (twitter/IRC) ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Avoid testStressTest Failing in Jenkins VDSM Unit Tests Job
On 08/09/2012 05:46 AM, Robert Middleswarth wrote: On 08/08/2012 10:37 PM, Zhou Zheng Sheng wrote: Hi all, Recently the oVirt Jenkins runs unit test for every new patch set in Gerrit. There are a lot of new patch sets every day, so the Jenkins may run the unit tests in parallel. I notice that most of the unit tests can be run in parallel except testStressTest. testStressTest creates lots of threads nearly to the system limit, so if we run two or more testStressTest, it will fail, giving false positives. We found this when we started running 5 of them at once. I have since limited the job to one per host. To stop the failures. I though I had resubmitted all the jobs that failed because of that. So I suggest the oVirt Jenkins run most of the tests in parellel, but run testStressTest exlusively. With the help of the Exclusion-Plugin, it its possible to configure Jenkins to run some steps in parellel while some steps exclusive in one job. Firstly, add a resource with the help of Exclusion-Plugin. Give a meaningful name to that resource, and assign the resource to this Job. Secondly, add a build step of Execute Shell, in this step, run all the tests other than testStressTest. So the shell script can be as follow. cd tests NOSE_EXCLUDE=testStressTest \ ./run_tests_local.sh \ --with-xunit --xunit-file=nosetests0.xml \ *.py Then add a build step of Critical Block Start. Then add a build step of Execute Shell, in this step, only run testStressTest as follow. cd tests ./run_tests_local.sh -m testStressTest \ --with-xunit --xunit-file=nosetests1.xml \ resourceManagerTests.py Then add a build step of Critical BlockEnd. At last, in post-build actions, in the Publish Test Result Report section, modify the test report XMLs to tests/nosetests*.xml. Jenkins will merge the test reports. Some patch sets are marked as verify failure by oVirt Jenkins. Most of them fails in testStressTest. I hope this can help the oVirt Jenkins a little bit. This information is useful to know. I wonder if we could just remove the stress test from the patch test and just leave it on the master branch tests. or you could do a separate job running only this test separate from all the other tests, and limit this job to one per host. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
