Re: [Openstack] Jenkins and transient failures
On Sun, Jul 01, 2012 at 08:40:36AM -0700, James E. Blair wrote: [snip] So with all that background, I think we should discuss the following at the CI team meeting on Tuesday: [snip] 3) Decide on a course of action to mitigate failures from transient gerrit errors (but continue to work on eliminating them in the first place). 4) Decide how to implement retriggering with Zuul. Can you expand on what you mean by this 4th point ? Is this a way to allow individual patch submitters to re-trigger builds on their own patches ? IIUC, currently only core reviewers can directly retrigger builds. It seems patch submitters are working around this restriction by simply doing no-op rebases re-uploading their patches again and again until Jenkins passes. If there's an easy way to allow re-triggering of failed builds by any any reviewer, it seems that would mitigate the pain of these failures, because we won't have to rely on a small pool of people to notice bogus failures. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins failure
Hi, Does anyone know what can cause the failure below: Started by userOpenStack Hudson https://jenkins.openstack.org/user/hudson-openstack [EnvInject] - Preparing an environment for the build. Building remotely onprecise6 https://jenkins.openstack.org/computer/precise6 in workspace /home/jenkins/workspace/gate-quantum-merge [gate-quantum-merge] $ /bin/bash -xe /tmp/hudson8164016237338935417.sh + /usr/local/jenkins/slave_scripts/gerrit-git-prep.sh review.openstack.org Triggered by:https://review.openstack.org/9203 + [[ ! -e .git ]] + git remote update Fetching origin + git reset --hard HEAD is now at c18822a OVS plugin support for v2 Quantum API + git clean -x -f -d -q + '[' -z '' ']' + git checkout master Already on 'master' Your branch is ahead of 'origin/master' by 1 commit. + git reset --hard remotes/origin/master HEAD is now at 4ac3207 Cisco's unplug_iface refers to non existing exception + git clean -x -f -d -q + '[' '!' -z '' ']' + merge_changes + set +x + merge_change openstack/quantum refs/changes/03/9203/1 + PROJECT=openstack/quantum + REFSPEC=refs/changes/03/9203/1 + git fetchhttps://review.openstack.org/p/openstack/quantum refs/changes/03/9203/1 error: The requested URL returned error: 403 while accessinghttps://review.openstack.org/p/openstack/quantum/info/refs fatal: HTTP request failed Build step 'Execute shell' marked build as failure Notifying upstream projects of job completion Finished: FAILURE Thanks in advance Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins/Gerrit
Hi, Anyone encountered any problems. When I try to access the server (https://review.openstack.org/) I get: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request /GET / https://review.openstack.org//. Reason: *Error reading from remote server* Apache/2.2.20 (Ubuntu) Server at review.openstack.org Port 443 Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins and transient failures
Kevin L. Mitchell kevin.mitch...@rackspace.com writes: One of the things that's really bugging me these days is transient failures, such as the inability to download a package, causing a gate job to fail. It seems to me that we can distinguish test failure from environment build failure easily enough, and automatically retry in the latter case. Is this possible in practice with our current CI infrastructure? Yes, that's certainly been a big annoyance lately. That's a good suggestion, though there are a couple of things that make it not straightforward: jenkins doesn't have a facility to easily express (through some means such as an exit code) that a job has had anything other than a simple success/failure outcome; I believe that's an open feature request with jenkins. Even if we worked around that, for better or worse since we started using virtualenv's instead of packages, a lot of what we're testing now includes things like dependencies, configuration, installation, and other items that are ancillary to unit tests themselves. If a change adds blorgh==1.0 to pip-requires, is the inability to install that a transient or permanent error? These may be solvable problems, but they'll take some engineering effort, and I have some ideas of where we may get a better return for our work. Most of the transient failures can be attributed to two causes: failures downloading packages, and failures connecting to gerrit. Monty has been working on a pypi mirror setup so that we can be responsible for ensuring that all of the python packages that pip needs to install are available to the jenkins slaves. We had hoped that simply adding a mirror would be enough, but as long as pip knows about both pypi.openstack.org and pypi.python.org, it will end up crawling the web pages of projects listed in the pypi mirror looking for new versions. So to really get to the point where we can run jobs with no unnecessary network dependencies, we have to be sure that our pypi mirror has every package needed, including when new dependencies are added. At the design summit, it was decided that we should move to a global list of dependencies for OpenStack -- with that in place, it should be easy to maintain the package inventory for our pypi mirror -- we can update the mirror when changes to the global dependency list are merged. However, that work seems to be stalled: https://bugs.launchpad.net/openstack-ci/+bug/995607 A reason we've seen even more errors downloading packages in recent weeks is that there have been some flaws with our pypi mirror implementation. Monty has been working this weekend to rectify those, so hopefully we'll see a significant drop in these errors when that is finished. And finally, as we've increased the number of builds jenkins runs (in order to run tests on new patchsets when they are uploaded, as well as run merge gates in parallel (which sometimes requires multiple runs of tests)) we have increased the load on the gerrit server which occasionally results in transient errors. Tuning gerrit is a bit of a black art; there's plenty of capacity on the server, but I believe further tuning is going to require a bit more instrumentation than we have now. Clark Boylan has been working on adding Java Melody to gerrit to help with that, so I hope we can get a handle on that soon. In the mean time, we have some ideas about how to work around that (retry with exponential backoff in the git scripts that jenkins uses, or cloning directly from a git repo instead of via gerrit). So with all that background, I think we should discuss the following at the CI team meeting on Tuesday: 1) What's the status of the global dependency list? Can we update: https://bugs.launchpad.net/openstack-ci/+bug/995607 Can we get it implemented in a reasonable amount of time to address these other issues (perhaps a couple of weeks)? 2) If not, can we make the pypi mirror be the only source of python packages for jenkins sooner? When we used pip bundles with tox, we set up the jobs to use the bundle unless there was a change to a -requires file. Could we do something similar and make pypi.openstack.org the only pip mirror unless there is a dependency change being tested? 3) Decide on a course of action to mitigate failures from transient gerrit errors (but continue to work on eliminating them in the first place). 4) Decide how to implement retriggering with Zuul. It's my very strong belief that our build systems should be robust enough that we don't need to retrigger jobs because of transient failures. It is not a good use of the time of busy and skilled developers to babysit jenkins jobs and retry them if the fail. So I think our priority should always be eliminating the causes of those failures, which is why I listed items 1-3 above in that order. However, there are always likely to be new causes for transient failures, and while we work on correcting them, we shouldn't make retrying builds any harder
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On 06/28/2012 11:58 PM, Monty Taylor wrote: On 06/28/2012 01:49 PM, Dan Prince wrote: - Original Message - From: Monty Taylor mord...@inaugust.com To: openstack@lists.launchpad.net Sent: Thursday, June 28, 2012 11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. I would add that if anyone else is interested in collaborating on making SmokeStack better I'm more than happy to give access. Its all open source and has been since Cactus. As is now SmokeStack can can cast a -1 vote and hopefully this is proving to be useful. I'm open to suggestions. I think it's stellar! The tests that smokestack runs could all be written such that they are run by jenkins. I actually put in quite a bit of work to maintain an openstack_vpc job on Jenkins post-Cactus. When we talked about gating on this job at the Diablo conference the idea didn't seem to get very far... I kind of saw that as the end of the line for maintaining an openstack_vpc job and eventually it went away. Not sure who deleted it, but anyway. The way I see it there is value in both testing systems. Rather than complaining about why one system exists and/or doesn't port its tests to the other why don't we build on each others strengths. Seeing a green verified +1 from both Jenkins and SmokeStack on a review should be very encouraging... and if one of the two systems fails it might require further investigation. I completely agree with that. I'm still hoping we'll see more systems from more people so that the set of combinations get larger. I think also there's clearly value in running tests, like how SmokeStack is doing right now, that aren't necessarily part of the gate, but which pro-actively provide useful information to the reviewers. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. The ability for developers to test things locally is very important. For that matter SmokeStack all started with a project called openstack_vpc, a project to spin up groups of cloud servers installed with the latest OpenStack code. A developer can use a project like openstack_vpc to spin up a set of servers in the cloud which builds and installs custom built packages for a set
[Openstack] Jenkins and transient failures
One of the things that's really bugging me these days is transient failures, such as the inability to download a package, causing a gate job to fail. It seems to me that we can distinguish test failure from environment build failure easily enough, and automatically retry in the latter case. Is this possible in practice with our current CI infrastructure? -- Kevin L. Mitchell kevin.mitch...@rackspace.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? Obviously this was all made worse by the transient problems we've had with the tests suite infrastructure these past 2 days, but regardless it seems like we have a gap in our merge approval procedures here. IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs to ensure tests are run on suitable hosts like SmokeStack does, or both. Regards, Daniel [1] -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. The tests that smokestack runs could all be written such that they are run by jenkins. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. Problem arise in paradise though. libvirt being one of them. It's not possible to install libvirt into a virtualenv, because it's a swig-based module built as part of the libvirt source itself. One of the solutions to this is to allow the testing virtual environments to use packages installed at the system level. We suggested this a little while ago, but this was rejected by the nova team who valued the benefit of having a restricted test run so that we know we've got all of the depends properly specified. To that end, after chatting with Brian Waldon, I put this up as a possible next try: https://review.openstack.org/#/c/8949/ Which adds an additional testing environment that has system software enabled and also installs additional optional things. With that environment, we should be able to run a jenkins gate that tests things with full libvirt, and also tests the mysql upgrade paths, without screwing our fine friends who run OSX. Fundamentally though - we're at a point of trying to have our cake and eat it too. Either we want comprehensive testing of all of the unit tests, or we want to be careful about not making the test environment to hard for a developer to exactly mimic. I'm obviously on the side of having us have gating tests that some devs might not be able to do on their laptops - such as running the libvirt tests properly. We're working on cloud software - worst case scenario if there's an intractable problem, as dev can always spin up an ubuntu image somewhere. Obviously this was all made worse by the transient problems we've had with the tests suite infrastructure these past 2 days, but regardless it seems like we have a gap in our merge approval procedures here. IMHO, either SmokeStack needs to be made compulsory, or Jenkins needs to ensure tests are run on suitable hosts like SmokeStack does, or both. The second is much more possible and as I've pointed out is in work - but I do think we should develop a clear sense that it's important to us that we
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On Thu, Jun 28, 2012 at 08:13:28AM -0700, Monty Taylor wrote: On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. The tests that smokestack runs could all be written such that they are run by jenkins. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. Ok, this makes sense to me now. I had assumed SmokeStack was a core part of the infrastructure. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. Problem arise in paradise though. libvirt being one of them. It's not possible to install libvirt into a virtualenv, because it's a swig-based module built as part of the libvirt source itself. One of the solutions to this is to allow the testing virtual environments to use packages installed at the system level. We suggested this a little while ago, but this was rejected by the nova team who valued the benefit of having a restricted test run so that we know we've got all of the depends properly specified. To that end, after chatting with Brian Waldon, I put this up as a possible next try: https://review.openstack.org/#/c/8949/ Which adds an additional testing environment that has system software enabled and also installs additional optional things. With that environment, we should be able to run a jenkins gate that tests things with full libvirt, and also tests the mysql upgrade paths, without screwing our fine friends who run OSX. Fundamentally though - we're at a point of trying to have our cake and eat it too. Either we want comprehensive testing of all of the unit tests, or we want to be careful about not making the test environment to hard for a developer to exactly mimic. I'm obviously on the side of having us have gating tests that some devs might not be able to do on their laptops - such as running the libvirt tests properly. We're working on cloud software - worst case scenario if there's an intractable problem, as dev can always spin up an ubuntu image somewhere. I think I agree with you, since in practice I believe the additional requirements on developers are not unreasonable in general. Taking the livirt example (though it applies to other examples like MySQL too)... If a developer is submitting changes that touches a part of OpenStack unrelated to the virt drivers, then there's low liklihood that they'll cause libvirt test failures. This means they won't need to have libvirt available themselves in the common case, and thus there's no new onerous requirement placed on them. A similar scenario occurs if they are touching non-libvirt drivers (eg XenAPI, VMWare). Only if they touch the libvirt driver itself, or things that it is derived /from/ (eg the base virt driver code), then are they likely to need to run the tests locally to toubleshoot. In such a case, it is not unreasonable that they be prepared to setup local installs for troubleshooting purposes. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On 06/28/2012 11:13 AM, Monty Taylor wrote: snip Fundamentally though - we're at a point of trying to have our cake and eat it too. Either we want comprehensive testing of all of the unit tests, or we want to be careful about not making the test environment to hard for a developer to exactly mimic. I'm obviously on the side of having us have gating tests that some devs might not be able to do on their laptops - such as running the libvirt tests properly. We're working on cloud software - worst case scenario if there's an intractable problem, as dev can always spin up an ubuntu image somewhere. I'm definitely in agreement here. Something as fundamental as libvirt is to openstack really needs to see testing in the main test environment. I really feel like this is almost the same conversation we had on IRC on Friday about the mysql requirement. There is a deeper level of testing that we can do if we start requiring more of the base OS in the jenkins environment. That takes us out of a place where all the unit tests that run in jenkins can be run in any python environment, but that's ok. More validation here on the gate is worth that. Maybe we need something that's different than the current @skip semantics, some @skip_unless_gate (or similar), because a big piece of this is also confusion about what the gate is actually testing, and what it's skipping. As the person involved in the patch that slipped through, I was as surprised as anyone else that it landed, but had an issue in a real libvirt test case. -Sean -- Sean Dague IBM Linux Technology Center email: sda...@linux.vnet.ibm.com alt-email: slda...@us.ibm.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On Jun 28, 2012, at 8:13 AM, Monty Taylor wrote: Which adds an additional testing environment that has system software enabled and also installs additional optional things. With that environment, we should be able to run a jenkins gate that tests things with full libvirt, and also tests the mysql upgrade paths, without screwing our fine friends who run OSX. Just fyi, libvirt is installable on OSX with brew and the tests currently run and pass. Vish___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On 06/28/2012 12:05 PM, Vishvananda Ishaya wrote: On Jun 28, 2012, at 8:13 AM, Monty Taylor wrote: Which adds an additional testing environment that has system software enabled and also installs additional optional things. With that environment, we should be able to run a jenkins gate that tests things with full libvirt, and also tests the mysql upgrade paths, without screwing our fine friends who run OSX. Just fyi, libvirt is installable on OSX with brew and the tests currently run and pass. Great! As a follow up - the full env got merged today. I'll add jenkins jobs to start running it. If you want to give it a spin locally, try: tox -v -efull And it should both install MySQL-python and set up the virtualenv with system-site-packages turned to true, so libvirt tests should run and not skip. In theory. In practice, it's virtualenv, so all bets are off. ;) Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
- Original Message - From: Monty Taylor mord...@inaugust.com To: openstack@lists.launchpad.net Sent: Thursday, June 28, 2012 11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. I would add that if anyone else is interested in collaborating on making SmokeStack better I'm more than happy to give access. Its all open source and has been since Cactus. As is now SmokeStack can can cast a -1 vote and hopefully this is proving to be useful. I'm open to suggestions. The tests that smokestack runs could all be written such that they are run by jenkins. I actually put in quite a bit of work to maintain an openstack_vpc job on Jenkins post-Cactus. When we talked about gating on this job at the Diablo conference the idea didn't seem to get very far... I kind of saw that as the end of the line for maintaining an openstack_vpc job and eventually it went away. Not sure who deleted it, but anyway. The way I see it there is value in both testing systems. Rather than complaining about why one system exists and/or doesn't port its tests to the other why don't we build on each others strengths. Seeing a green verified +1 from both Jenkins and SmokeStack on a review should be very encouraging... and if one of the two systems fails it might require further investigation. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. The ability for developers to test things locally is very important. For that matter SmokeStack all started with a project called openstack_vpc, a project to spin up groups of cloud servers installed with the latest OpenStack code. A developer can use a project like openstack_vpc to spin up a set of servers in the cloud which builds and installs custom built packages for a set of Git URLs. So essentially the underpinnings of SmokeStack *can* all be done from a local machine just like they run from the UI. There is also value in testing things differently in ways which may not be easy for all developers to reproduce. Take XenServer for example... not every developer
Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers
On 06/28/2012 01:49 PM, Dan Prince wrote: - Original Message - From: Monty Taylor mord...@inaugust.com To: openstack@lists.launchpad.net Sent: Thursday, June 28, 2012 11:13:28 AM Subject: Re: [Openstack] Jenkins vs SmokeStack tests Gerrit merge blockers On 06/28/2012 07:32 AM, Daniel P. Berrange wrote: Today we face a situation where Nova GIT master fails to pass all the libvirt test cases. This regression was accidentally introduced by the following changeset https://review.openstack.org/#/c/8778/ If you look at the history of that, the first SmokeStack test run fails with some (presumably) transient errors, and added negative karma to the change against patchset 2. If it were not for this transient failure, it should have shown the regression in the libvirt test case. The libvirt test case in question was one that is skipped, unless libvirt is actually present on the host running the tests. SmokeStack had made sure the tests would run on such a host. There were then further patchsets uploaded, and patchset 4 was approved for merge. Jenkins ran its gate jobs and these all passed successfully. I am told that Jenkins will actually run the unittests that are included in Nova, so I would have expected it to see the flawed libvirt test case, but it didn't. I presume therefore, that Jenkins is not running on a libvirt enabled host. Kind of - it's sadly more complex than that ... The end result was that the broken changeset was merged to master, which in turns means any other developers submitting changes touching the libvirt area will get broken tests reported that were not actually their own fault. This leaves me with the following questions... 1. Why was the recorded failure from SmokeStack not considered to be a blocker for the merge of the commit by Gerrit or Jenkins or any of the reviewers ? 2. Why did SmokeStack not get re-triggered for the later patch set revisions, before it was merged ? The answer to 1 and 2 is largely the same - SmokeStack is a community contributed resources and is not managed by the CI team. Dan Prince does a great job with it, but it's not a resource that we have the ability to fix should it start messing up, so we have not granted it the permissions to file blocking votes. I would add that if anyone else is interested in collaborating on making SmokeStack better I'm more than happy to give access. Its all open source and has been since Cactus. As is now SmokeStack can can cast a -1 vote and hopefully this is proving to be useful. I'm open to suggestions. I think it's stellar! The tests that smokestack runs could all be written such that they are run by jenkins. I actually put in quite a bit of work to maintain an openstack_vpc job on Jenkins post-Cactus. When we talked about gating on this job at the Diablo conference the idea didn't seem to get very far... I kind of saw that as the end of the line for maintaining an openstack_vpc job and eventually it went away. Not sure who deleted it, but anyway. The way I see it there is value in both testing systems. Rather than complaining about why one system exists and/or doesn't port its tests to the other why don't we build on each others strengths. Seeing a green verified +1 from both Jenkins and SmokeStack on a review should be very encouraging... and if one of the two systems fails it might require further investigation. I completely agree with that. I'm still hoping we'll see more systems from more people so that the set of combinations get larger. I think also there's clearly value in running tests, like how SmokeStack is doing right now, that aren't necessarily part of the gate, but which pro-actively provide useful information to the reviewers. The repos that run the jenkins tests are all in git and managed by openstack's gerrit. If there are testing profiles that it runs that we as a community value and want to see part of the gate, anyone is welcome to port them. 3. Why did Jenkins not ensure that the tests were run on a libvirt enabled host ? This is a different, and slightly more complex. We run tests in virtualenvs so that the process used to test the code can be consistently duplicated by all of the developers in the project. This is the reason that we no longer do ubuntu package creation as part of the gate - turns out that's really hard for a developer running on OSX to do locally on their laptop - and if Jenkins reports an blocking error in a patch, we want a developer to be able to reproduce the problem locally so that they can have a chance at fixing it. The ability for developers to test things locally is very important. For that matter SmokeStack all started with a project called openstack_vpc, a project to spin up groups of cloud servers installed with the latest OpenStack code. A developer can use a project like openstack_vpc to spin up a set of servers in the cloud
[Openstack] Jenkins
Hi, Is anyone aware of a problem with Jenkins? Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins
Gary, Apparently, it happened to me. Tests took 13 hours and came with failure. Thanks, Joseph (w) 703-248-6160 (f) 703-812-3712 3811 N. Fairfax Drive Suite 200 Arlington, VA, 22203, USA http://www.east.isi.edu/~jsuh - Original Message - From: Gary Kotton gkot...@redhat.com To: openstack@lists.launchpad.net Sent: Wednesday, June 27, 2012 1:04:28 PM Subject: [Openstack] Jenkins Hi, Is anyone aware of a problem with Jenkins? Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins
It had some problems… now it just seems backed up… trying to catch up. On Jun 27, 2012, at 12:04 PM, Gary Kotton wrote: Hi, Is anyone aware of a problem with Jenkins? Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins
Yes, it is currently being worked on... John Postlethwait Nebula, Inc. On Wednesday, June 27, 2012 at 10:04 AM, Gary Kotton wrote: Hi, Is anyone aware of a problem with Jenkins? Thanks Gary ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net (mailto:openstack@lists.launchpad.net) Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins Changes
There appears to have been some changes to Jenkins recently. Jobs do not appear to be associated with Gerrit changes anymore. https://jenkins.openstack.org/job/gate-nova-python27/ You can see jobs 335 and older have a link to Gerrit, but new jobs don't have any. This makes it harder to see what is happening in Jenkins but also removes the Retrigger link I used to use when there was a transient failure in a job. Was this an intended change in behavior? JE ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Changes
Johannes Erdfelt johan...@erdfelt.com writes: There appears to have been some changes to Jenkins recently. Jobs do not appear to be associated with Gerrit changes anymore. https://jenkins.openstack.org/job/gate-nova-python27/ You can see jobs 335 and older have a link to Gerrit, but new jobs don't have any. This makes it harder to see what is happening in Jenkins but also removes the Retrigger link I used to use when there was a transient failure in a job. Was this an intended change in behavior? Yes, see the message with subject Parallel execution of Jenkins gate jobs on this list. The short version is that we've moved most triggering of Jenkins jobs out of the Gerrit Trigger Plugin. The bad news about that is that we lose the those links and the retrigger buttons. The good news is that we get parallel execution of gate jobs. This means that if you approve, say, four changes to nova, they will all test and merge at the same time (if they all succeed). Especially when we start gating on tempest, we are looking at significantly longer run-times for gate tests, so we very much need the ability to increase the rate at which we can test and merge changes. I think the trade off will be worth it. You can still link Jenkins jobs to Gerrit changes though. Near the top of most jobs, you will see a line like: Triggered by: https://review.openstack.org/8228 So you can see what change triggered a Job. Because jobs are run and are reported more quickly (Jenkins has gone from having a _huge_ queue of jobs to run to generally having no queue) the time between when there have been jobs triggered and when they are reported to Gerrit should be shorter, so there should be less need to search for a job in Jenkins. I also miss the retrigger button. If it is very important, we can work on adding equivalent functionality back. However, core members can always retrigger a change by leaving another Approved vote. Additionally, we're working on reducing transient failures -- we've just rolled out a local pypi mirror that is built from the requirements of the collection of projects. Hopefully we can reduce the number of transient failures and we won't need that functionality. -Jim ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins moved to new server
Hi, The server that was hosting Jenkins was getting a bit long in the tooth, so this weekend we moved Jenkins to a new one. It is now running on a cloud server with more CPU and memory so it should be a little more responsive. It was built using our puppet module[1], so we have an actual standardized install process and it should be much easier to manage in the future. All the job history was copied over, so there shouldn't be any visible changes. We did take the opportunity to upgrade to the latest jenkins along with all the plugins. We haven't seen any serious issues so far, but please let us know if you notice something behaving differently. -Jim [1] https://github.com/openstack/openstack-ci-puppet ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] jenkins: nova-tarmac hangs
On 08/19/2011 10:09 AM, Dan Prince wrote: The nova-tarmac jobs gets hung quite often lately. I know a lot of fingers have been pointing at Launchpad issues but I'm not sure that is the case here. When its hung the console output of the job is almost always at: Running test command: /home/jenkins/openstack-ci/test_nova.sh This is essentially runs: 'bash run_tests.sh -N python setup.py sdist'. --- I've also noticed that the nova-coverage job is also often running at the same time. This job also appears to use 'run_tests.sh -N'. Additionally this job like nova-tarmac also runs on the 'nova' jenkins slave. Since both these jobs use the 'nova' Jenkins slave I think the problem here is we can't have two jobs using 'run_tests.sh -N' at the same time (due to potential port issues etc). Should we create isolated build slaves for these jobs and/or prevent them from running at the same time? Just to follow up to the list, since we addressed this on IRC... We have now restricted the nova build slave to only have one executor at a time - and then we added another build slave. We'll see if this takes care of things. Monty ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] jenkins: nova-tarmac hangs
The nova-tarmac jobs gets hung quite often lately. I know a lot of fingers have been pointing at Launchpad issues but I'm not sure that is the case here. When its hung the console output of the job is almost always at: Running test command: /home/jenkins/openstack-ci/test_nova.sh This is essentially runs: 'bash run_tests.sh -N python setup.py sdist'. --- I've also noticed that the nova-coverage job is also often running at the same time. This job also appears to use 'run_tests.sh -N'. Additionally this job like nova-tarmac also runs on the 'nova' jenkins slave. Since both these jobs use the 'nova' Jenkins slave I think the problem here is we can't have two jobs using 'run_tests.sh -N' at the same time (due to potential port issues etc). Should we create isolated build slaves for these jobs and/or prevent them from running at the same time? Dan ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Jenkins Job Configuration
Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_smith=dell@lists.launchpad.net [mailto:openstack-bounces+adrian_f_smith=dell@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files ... but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files ... but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
That being said - I'm also more than happy to: a) have people help admin the jenkins and/or b) work with people to get jobs they want added to the jenkins so that replicating it elsewhere isn't needed. Of course, getting the config published is still important... but we can also work on getting you access if that makes sense. Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
Hi, there is a plugin for Jenkins that extends standart permission matrix. http://wiki.hudson-ci.org/display/HUDSON/Extended+Read+Permission+Plugin Maybe this would help. On Thu, Jul 14, 2011 at 7:27 PM, adrian_f_sm...@dell.com wrote: That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_smith=dell@lists.launchpad.net[mailto: openstack-bounces+adrian_f_smith=dell@lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files ... but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Best regards, Alexander Sakhnov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
YOU ARE MY PERSONAL HERO! On 07/14/2011 11:50 AM, Alexander Sakhnov wrote: Hi, there is a plugin for Jenkins that extends standart permission matrix. http://wiki.hudson-ci.org/display/HUDSON/Extended+Read+Permission+Plugin Maybe this would help. On Thu, Jul 14, 2011 at 7:27 PM, adrian_f_sm...@dell.com mailto:adrian_f_sm...@dell.com wrote: That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_smith=dell.com http://dell.com@lists.launchpad.net http://lists.launchpad.net [mailto:openstack-bounces+adrian_f_smith mailto:openstack-bounces%2Badrian_f_smith=dell.com http://dell.com@lists.launchpad.net http://lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Best regards, Alexander Sakhnov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
This has now been installed and activated. Anonymous users should be able to read the config of any job now. I've got admin access, so it's kinda of hard to test - would someone mind verifying that they can, in fact, see the job configs? Thanks! Monty On 07/14/2011 11:50 AM, Alexander Sakhnov wrote: Hi, there is a plugin for Jenkins that extends standart permission matrix. http://wiki.hudson-ci.org/display/HUDSON/Extended+Read+Permission+Plugin Maybe this would help. On Thu, Jul 14, 2011 at 7:27 PM, adrian_f_sm...@dell.com mailto:adrian_f_sm...@dell.com wrote: That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_smith=dell.com http://dell.com@lists.launchpad.net http://lists.launchpad.net [mailto:openstack-bounces+adrian_f_smith mailto:openstack-bounces%2Badrian_f_smith=dell.com http://dell.com@lists.launchpad.net http://lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Best regards, Alexander Sakhnov ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
Works great! On Thu, Jul 14, 2011 at 12:35 PM, Monty Taylor mord...@inaugust.com wrote: This has now been installed and activated. Anonymous users should be able to read the config of any job now. I've got admin access, so it's kinda of hard to test - would someone mind verifying that they can, in fact, see the job configs? Thanks! Monty On 07/14/2011 11:50 AM, Alexander Sakhnov wrote: Hi, there is a plugin for Jenkins that extends standart permission matrix. http://wiki.hudson-ci.org/**display/HUDSON/Extended+Read+** Permission+Pluginhttp://wiki.hudson-ci.org/display/HUDSON/Extended+Read+Permission+Plugin Maybe this would help. On Thu, Jul 14, 2011 at 7:27 PM, adrian_f_sm...@dell.com mailto:Adrian_F_Smith@dell.**com adrian_f_sm...@dell.com wrote: That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_**smith=dell.com http://dell.com@lists.**launchpad.net http://lists.launchpad.net http://lists.launchpad.net [mailto:openstack-bounces+**adrian_f_smithopenstack-bounces%2Badrian_f_smith mailto:openstack-bounces%**2Badrian_f_smithopenstack-bounces%252Badrian_f_smith =dell.com http://dell.com@lists.**launchpad.net http://lists.launchpad.net http://lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.net mailto:openstack@lists.** launchpad.net openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.**launchpad.netopenstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.**launchpad.netopenstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.**launchpad.netopenstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp -- Best regards, Alexander Sakhnov __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp __**_ Mailing list: https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~**openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/**ListHelphttps://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins Job Configuration
Configs are visible to me too. Regards, Rohit From: openstack-bounces+rohit.karajgi=vertex.co...@lists.launchpad.net [mailto:openstack-bounces+rohit.karajgi=vertex.co...@lists.launchpad.net] On Behalf Of Josh Kearney Sent: Thursday, July 14, 2011 11:40 PM To: Monty Taylor Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Works great! On Thu, Jul 14, 2011 at 12:35 PM, Monty Taylor mord...@inaugust.commailto:mord...@inaugust.com wrote: This has now been installed and activated. Anonymous users should be able to read the config of any job now. I've got admin access, so it's kinda of hard to test - would someone mind verifying that they can, in fact, see the job configs? Thanks! Monty On 07/14/2011 11:50 AM, Alexander Sakhnov wrote: Hi, there is a plugin for Jenkins that extends standart permission matrix. http://wiki.hudson-ci.org/display/HUDSON/Extended+Read+Permission+Plugin Maybe this would help. On Thu, Jul 14, 2011 at 7:27 PM, adrian_f_sm...@dell.commailto:adrian_f_sm...@dell.com mailto:adrian_f_sm...@dell.commailto:adrian_f_sm...@dell.com wrote: That something will likely be a drop of a bunch of xml files That would be perfect. Thanks Monty. Adrian -Original Message- From: openstack-bounces+adrian_f_smith=dell.comhttp://dell.com http://dell.com@lists.launchpad.nethttp://lists.launchpad.net http://lists.launchpad.net [mailto:openstack-bounces+adrian_f_smithmailto:openstack-bounces%2Badrian_f_smith mailto:openstack-bounces%2Badrian_f_smithmailto:openstack-bounces%252Badrian_f_smith=dell.comhttp://dell.com http://dell.com@lists.launchpad.nethttp://lists.launchpad.net http://lists.launchpad.net] On Behalf Of Monty Taylor Sent: Thursday, July 14, 2011 4:01 PM To: openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Subject: Re: [Openstack] Jenkins Job Configuration Hi! We're working on a decent solution for this. Jenkins itself does not have a setting which allows you to see the job config without also giving you access to edit it. (fail) However, it's been on my todo list to a) just publish these somewhere or b) even better, an easy way for you to spin up an identical jenkins (albeit one which does not publish tarballs) I'm going to move a) up on my list and see if I can't get you something today. That something will likely be a drop of a bunch of xml files but it's at least something. :) Monty On 07/14/2011 09:30 AM, adrian_f_sm...@dell.commailto:adrian_f_sm...@dell.com wrote: Would it be possible to see the individual configuration files for the jobs running on http://jenkins.openstack.org? ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp -- Best regards, Alexander Sakhnov ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.netmailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstackhttps://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net
[Openstack] Jenkins - nova-pylint-errors
I've been working on cleaning up pylint errors and have a concern regarding:
E1101: %s %r has no %r member
Used when a variable is accessed for an unexistent member.
97 (59%) of the 169 errors in the nova pylint errors job on jenkins are of the
type E1101. 82/97 are located under the sqlalchemy directory. This is (mostly)
because pylint doesn't understand most of the ways classes and functions in the
database code are getting members even though they are by no means incorrect.
One example that pylint complains about is this:
nova/db/sqlalchemy/api.py
Class 'Network' has no 'fixed_ips' member
1322@require_admin_context
1323def network_get_by_instance(_context, instance_id):
1324session = get_session()
1325rv = session.query(models.Network).\
1326 filter_by(deleted=False).\
! 1327 join(models.Network.fixed_ips).\
1328 filter_by(instance_id=instance_id).\
1329 filter_by(deleted=False).\
1330 first()
But Network does have the attribute fixed_ips because it is created here:
nova/db/sqlalchemy/models.py
536: network = relationship(Network, backref=backref('fixed_ips'))
What do you guys think about setting jenkins to ignore E1101 or at least ignore
it for nova/db/sqlalchemy? If we ever want to get our pylint errors down to
zero so we can start enforcing it, we need to either do something like above or
insert pylint ignore statements throughout all the code where the error is not
really an error.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Jenkins - nova-pylint-errors
Hey Vish,
I may be missing something but when I test with pylint locally, I am using the
pylintrc located in trunk. The only cleanup I am doing is derived directly from
looking at the nova-pylint-errors job on Jenkins.
Alex
-Original Message-
From: Vishvananda Ishaya vishvana...@gmail.com
Sent: Monday, June 6, 2011 11:42am
To: Alex Meade alex.me...@rackspace.com
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Jenkins - nova-pylint-errors
Hey alex,
Justin came up with a pretty reasonable set of defaults for pylint that tends
to only show real error messages. It is the nova-pylint-errors job in jenkins.
It looks like this error has already been removed from the list (although
there is another one there about medatada #690)
Perhaps you should start your cleanup by fixing the ones still showing up in
that job rather than the default pylint output?
Vish
On Jun 6, 2011, at 6:08 AM, Alex Meade wrote:
I've been working on cleaning up pylint errors and have a concern regarding:
E1101: %s %r has no %r member
Used when a variable is accessed for an unexistent member.
97 (59%) of the 169 errors in the nova pylint errors job on jenkins are of
the type E1101. 82/97 are located under the sqlalchemy directory. This is
(mostly) because pylint doesn't understand most of the ways classes and
functions in the database code are getting members even though they are by no
means incorrect.
One example that pylint complains about is this:
nova/db/sqlalchemy/api.py
Class 'Network' has no 'fixed_ips' member
1322@require_admin_context
1323def network_get_by_instance(_context, instance_id):
1324session = get_session()
1325rv = session.query(models.Network).\
1326 filter_by(deleted=False).\
! 1327 join(models.Network.fixed_ips).\
1328 filter_by(instance_id=instance_id).\
1329 filter_by(deleted=False).\
1330 first()
But Network does have the attribute fixed_ips because it is created here:
nova/db/sqlalchemy/models.py
536: network = relationship(Network, backref=backref('fixed_ips'))
What do you guys think about setting jenkins to ignore E1101 or at least
ignore it for nova/db/sqlalchemy? If we ever want to get our pylint errors
down to zero so we can start enforcing it, we need to either do something
like above or insert pylint ignore statements throughout all the code where
the error is not really an error.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
