On 1/15/2015 5:43 PM, Steven Hardy wrote:
On Thu, Jan 15, 2015 at 04:49:37PM -0600, Matt Riedemann wrote:
On 1/15/2015 11:40 AM, Matt Riedemann wrote:
On 1/13/2015 9:27 PM, Matt Riedemann wrote:
On 1/13/2015 12:11 PM, Steven Hardy wrote:
On Tue, Jan 13, 2015 at 10:00:04AM -0600, Matt Riedemann wrote:
Looks like the fix we merged didn't actually fix the problem. I have
a patch
[1] to uncap the boto requirement on master and it's failing the ec2
tests
in tempest the same as before.
FWIW, I just re-tested and boto 2.35.1 works fine for me locally, if you
revert my patch it breaks again with "Signature not provided" errors
(for
all ec2 API requests).
If you look at the failures in the log, it actually looks like a
different
problem:
EC2ResponseError: EC2ResponseError: 401 Unauthorized
This is not the same as the original error which rejected any request
inside the nova API before even calling keystone with a message like
this:
AuthFailure: Signature not provided
AFAICT this means my patch is working, and there's a different problem
affecting only a subset of the ec2 boto tests.
Steve
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Yeah, new bug reported, looks like we're hitting 401 Unauthorized errors
when trying to create security groups in the test:
https://bugs.launchpad.net/nova/+bug/1410622
I have a debug patch up here to try and recreate the tempest failures
with latest boto but using a nova debug change also to get more
information when we fail.
https://review.openstack.org/#/c/147601/
I finally narrowed this down to some code in keystone where it generates a
signature and compares that to what nova is passing in on the request for
ec2 credentials and they are different so keystone is rejecting the request
with a 401.
http://logs.openstack.org/01/147601/3/check/check-tempest-dsvm-full/96bb05e/logs/apache/keystone.txt.gz#_2015-01-15_22_00_27_046
I'm assuming something needs to change in keystone to support the version 4
format?
Keystone already supports the hmac v4 format, the code which creates the
signature keystone compares with that in the request actually lives in
keystoneclient:
https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L156
We have been bitten by a couple of bugs in the past where boto changed the
way it calculated the signature (presumably in a way compatible with AWS),
but generally that breaks all requests not just a specific one as in this
case (CreateSecurityGroup).
Interestingly, this seems to work OK for me locally, e.g using
euca-add-group, which AFAICS is the same API action the test is failing on:
-bash-4.2$ source /opt/stack/devstack/accrc/demo/demo
-bash-4.2$ euca-add-group --debug --description foo shtest2
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using access key provided by
client.
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using secret key provided by
client.
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Method: POST
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Path: /services/Cloud/
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Data:
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Headers: {}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Host: 192.168.0.4
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Port: 8773
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Params: {'Action':
'CreateSecurityGroup', 'GroupName': 'shtest2', 'Version': '2009-11-30',
'GroupDescription': 'foo'}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:establishing HTTP connection:
kwargs={'port': 8773, 'timeout': 70}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Token: None
2015-01-15 23:33:22,891 euca2ools [DEBUG]:CanonicalRequest:
POST
/services/Cloud/
host:192.168.0.4:8773
x-amz-date:20150115T233322Z
host;x-amz-date
a364b884b3e72160b8850f80c1b5b559011b38313da026d04dd60b745c0fd135
2015-01-15 23:33:22,892 euca2ools [DEBUG]:StringToSign:
AWS4-HMAC-SHA256
20150115T233322Z
20150115/168/192/aws4_request
6b751f8ad4ca1935c68a35f196928c1626d8fdb0ae4bf901050ab757649c9a9a
2015-01-15 23:33:22,892 euca2ools [DEBUG]:Signature:
3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31
2015-01-15 23:33:22,892 euca2ools [DEBUG]:Final headers: {'Content-Length':
'84', 'User-Agent': 'Boto/2.35.1 Python/2.7.5
Linux/3.17.7-200.fc20.x86_64', 'Host': '192.168.0.4:8773', 'X-Amz-Date':
'20150115T233322Z', 'Content-Type': 'application/x-www-form-urlencoded;
charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256
Credential=8943303abb374011be595587aa1fc986/20150115/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31'}
2015-01-15 23:33:22,994 euca2ools [DEBUG]:Response headers: [('date', 'Thu,
15 Jan 2015 23:33:22 GMT'), ('content-length', '407'), ('content-type',
'text/xml')]
2015-01-15 23:33:22,995 euca2ools [DEBUG]:<CreateSecurityGroupResponse
xmlns="http://ec2.amazonaws.com/doc/2009-11-30/";>
<requestId>req-cf3700c4-51fa-4d1f-bc6b-5ee45321a215</requestId>
<securityGroupSet>
<item>
<ipPermissions/>
<groupName>shtest2</groupName>
<groupDescription>foo</groupDescription>
<ownerId>e463cc1c9e814680bd96f1f87ca61e37</ownerId>
</item>
</securityGroupSet>
</CreateSecurityGroupResponse>
-bash-4.2$ pip freeze | grep boto
boto==2.35.1
So I guess the next step is to figure out what's different between the
request made by euca-add-group and the failing test (assuming others can
reproduce the successful call via euca-add-group that is..).
Steve
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
I went and checked out the separate ec2-api repo in stackforge and they
have a different way of handling the v4 signature (it's quite a bit more
code):
https://github.com/stackforge/ec2-api/commit/e4c4463ab10ebf6e21d84f9b3369494b0055f963
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
