Re: [openstack-dev] [All] Fixed recent gate issues
2014-02-23 10:52 GMT+01:00 Gary Kotton gkot...@vmware.com: It looks like this does not solve the issue. Yeah https://review.openstack.org/74451 doesn't solve the issue completely, we have SKIP_EXERCISES=boot_from_volume,bundle,client-env,euca,swift,client-args but failure is now in Grenade's Javelin script: + swift upload javelin /etc/hosts ...(same Traceback)... [ERROR] /opt/stack/new/grenade/setup-javelin:151 Swift upload failed I wonder if we need the same change for stable/havana. devstack-gate master branch handles all projects branches, patch above was inside if stable/grizzly Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
https://review.openstack.org/74451 doesn't solve the issue completely, we have SKIP_EXERCISES=boot_from_volume,bundle,client-env,euca,swift,client-args but failure is now in Grenade's Javelin script: + swift upload javelin /etc/hosts ...(same Traceback)... [ERROR] /opt/stack/new/grenade/setup-javelin:151 Swift upload failed What about just removing that test from Javelin in stable/havana? https://review.openstack.org/76058 Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
On 2/19/14 10:31 PM, Alan Pevec ape...@gmail.com wrote: Yeah it's pip weirdness where things falls apart because of version cap. It's basically installing bin/swift from 1.9 when it sees the version requirement but it leaves everything in python-swiftclient namespace from master. So I've actually been looking at this since late yesterday the conclusion we've reached is to just skip the exercises on grizzly. Removing the version cap isn't going to be simple on grizzly because there global requirements wasn't enforced back in grizzly. We'd have to change the requirement for both glance, horizon, and swift and being ~3 weeks away from eol for grizzly I don't think we should mess with that. This failure is only an issue with cli swiftclient on grizzly (and one swift functional test) which as it sits now is just the devstack exercises on grenade. So if we just don't run those exercises on the grizzly side of a grenade run there shouldn't be an issue. I've got 2 patches to do this here: https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/% 23/c/74419/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu% 2BfDtysg45MkPhCZFxPEq8%3D%0Am=PlTk6Hx5SymtM%2BlXLUIqyNGMz3z9JFKMcy7XJRvm k6s%3D%0As=591c1b105d86204158b21bd572a7f589f8ad032359282b84da9bc2b649d1a 3f9 https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/% 23/c/74451/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu% 2BfDtysg45MkPhCZFxPEq8%3D%0Am=PlTk6Hx5SymtM%2BlXLUIqyNGMz3z9JFKMcy7XJRvm k6s%3D%0As=d531e1933102045d5b77138fc42bd8e68ddc3d5b25dd281d94bfb0e9a22b9 a64 Looks like only the latter is needed, devstack-gate core please approve it to unblock stable/havana. It looks like this does not solve the issue. I wonder if we need the same change for stable/havana. Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi- bin/mailman/listinfo/openstack-devk=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=e H0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0Am=PlTk6Hx5SymtM%2BlXLUI qyNGMz3z9JFKMcy7XJRvmk6s%3D%0As=8a9c304e78b852d9eaa9063697282ccd38a303300 54aebe635797a8bce793bbe ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
Yeah it's pip weirdness where things falls apart because of version cap. It's basically installing bin/swift from 1.9 when it sees the version requirement but it leaves everything in python-swiftclient namespace from master. So I've actually been looking at this since late yesterday the conclusion we've reached is to just skip the exercises on grizzly. Removing the version cap isn't going to be simple on grizzly because there global requirements wasn't enforced back in grizzly. We'd have to change the requirement for both glance, horizon, and swift and being ~3 weeks away from eol for grizzly I don't think we should mess with that. This failure is only an issue with cli swiftclient on grizzly (and one swift functional test) which as it sits now is just the devstack exercises on grenade. So if we just don't run those exercises on the grizzly side of a grenade run there shouldn't be an issue. I've got 2 patches to do this here: https://review.openstack.org/#/c/74419/ https://review.openstack.org/#/c/74451/ Looks like only the latter is needed, devstack-gate core please approve it to unblock stable/havana. Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
Hi John, thanks for the summary. I've noticed one more fall out from swiftclient update in Grenade jobs running on stable/havana changes e.g. http://logs.openstack.org/02/73402/1/check/check-grenade-dsvm/a5650ac/console.html ... 2014-02-18 13:00:02.103 | Test Swift 2014-02-18 13:00:02.103 | + swift --os-tenant-name=demo --os-username=demo --os-password=secret --os-auth-url=http://127.0.0.1:5000/v2.0 stat 2014-02-18 13:00:02.284 | Traceback (most recent call last): 2014-02-18 13:00:02.284 | File /usr/local/bin/swift, line 35, in module 2014-02-18 13:00:02.284 | from swiftclient import Connection, HTTPException 2014-02-18 13:00:02.285 | ImportError: cannot import name HTTPException 2014-02-18 13:00:02.295 | + STATUS_SWIFT=Failed ... Grenade job installs swiftclient from git master but then later due to python-swiftclient=1.2,2 requirement in Grizzly, older version 1.9.0 is pulled from pypi and then half-installed or something, producing above conflict between swift CLI binary and libs. Solution could be to remove swiftclient cap in Grizzly, any other suggestions? Cheers, Alan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
On Tue, Feb 18, 2014 at 08:27:16PM +0100, Alan Pevec wrote: Hi John, thanks for the summary. I've noticed one more fall out from swiftclient update in Grenade jobs running on stable/havana changes e.g. http://logs.openstack.org/02/73402/1/check/check-grenade-dsvm/a5650ac/console.html ... 2014-02-18 13:00:02.103 | Test Swift 2014-02-18 13:00:02.103 | + swift --os-tenant-name=demo --os-username=demo --os-password=secret --os-auth-url=http://127.0.0.1:5000/v2.0 stat 2014-02-18 13:00:02.284 | Traceback (most recent call last): 2014-02-18 13:00:02.284 | File /usr/local/bin/swift, line 35, in module 2014-02-18 13:00:02.284 | from swiftclient import Connection, HTTPException 2014-02-18 13:00:02.285 | ImportError: cannot import name HTTPException 2014-02-18 13:00:02.295 | + STATUS_SWIFT=Failed ... Grenade job installs swiftclient from git master but then later due to python-swiftclient=1.2,2 requirement in Grizzly, older version 1.9.0 is pulled from pypi and then half-installed or something, producing above conflict between swift CLI binary and libs. Solution could be to remove swiftclient cap in Grizzly, any other suggestions? Yeah it's pip weirdness where things falls apart because of version cap. It's basically installing bin/swift from 1.9 when it sees the version requirement but it leaves everything in python-swiftclient namespace from master. So I've actually been looking at this since late yesterday the conclusion we've reached is to just skip the exercises on grizzly. Removing the version cap isn't going to be simple on grizzly because there global requirements wasn't enforced back in grizzly. We'd have to change the requirement for both glance, horizon, and swift and being ~3 weeks away from eol for grizzly I don't think we should mess with that. This failure is only an issue with cli swiftclient on grizzly (and one swift functional test) which as it sits now is just the devstack exercises on grenade. So if we just don't run those exercises on the grizzly side of a grenade run there shouldn't be an issue. I've got 2 patches to do this here: https://review.openstack.org/#/c/74419/ https://review.openstack.org/#/c/74451/ -Matt Treinish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [All] Fixed recent gate issues
On 02/14/2014 02:15 PM, John Dickinson wrote: As many of you surely noticed, we had some significant significant gate issues in the last day. It's fixed now, and I've got the details below. The root cause of the issue was a lack of proper testing in python- swiftclient. We've made some improvements here in the last few hours, but improving this will be a longer-term effort (and one that is being prioritized). Here's what happened: In order to get support for TLS certificate validation, swiftclient was ported to use the Python requests library. This is a good change, overall, but it was merged with a bug where the object data was uploaded as a multipart/form-data instead of as the raw data itself. This issue was resolved with patch https://review.openstack.org/#/c/73585/. The gate is currently stable, everyone should be unblocked by this issue now. If you have a patch that failed a check or gate run, you should recheck/reverify with bug #1280072. This, of course, raises the question of how this was allowed to happen. First, there is a lack of functional testing in swiftclient itself. (And, mea culpa, I should have done better testing before I merged the initial breaking patch.) These tests are being prioritized and worked on now. Second, python-swiftclient did not have a symmetric gate with the other projects that depend upon it. Although the gate change to make this happen was proposed quite a while ago, it wasn't merged until just this morning (https://review.openstack.org/#/c/70378/). Having these tests earlier should have caught the issues in the original python-swiftclient patch. Now that it has landed, there is much less risk of such a problem happening again. I want to thank Jeremy Stanley on the infra team for helping getting these patches landed quickly. I'd also like to thank Tristan Cacqueray for helping get the fixes written for python-swiftclient. --John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thank you for the comprehensive write up, John. Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
