Hi.
I am trying Heat instance HA, using RDO Icehouse.
After instance boot, instance push own stats to heat alarm with
cfn-push-stats command.
But cfn-push-stats always failed with error '403
SignatureDoesNotMatch', this message is
output to /var/log/cfn-push-stats.log.
I debugged client and server side code. (i.e. cfn-push-stats, boto,
heat, keystone,
keystoneclient) And I found curious code mismatch between boto and
keystoneclient about
signature calculation.
Here is a result of debugging, and code examination.
* Client side
cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST'
request with boto.
boto perfomes signature calculation. [1]
for signature calculation, firstly it construct 'CanonicalRequest',
some strings are joined.
And create a digest hash of the CanonicalRequest for signature
calculation.
CanonicalRequest contains CanonicalQueryString, which is transfomed
URL query strings.
CanonicalRequest =
HTTPRequestMethod + '\n' +
CanonicalURI + '\n' +
CanonicalQueryString + '\n' +
CanonicalHeaders + '\n' +
SignedHeaders + '\n' +
HexEncode(Hash(RequestPayload))
**AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as
CanonicalQueryString, when request is POST.**
AWS original tool's code is following.
cfnbootstrap/aws_client.py
110 class V4Signer(Signer):
144 (canonical_headers, signed_headers) =
self._canonicalize_headers(new_headers)
145 canonical_request += canonical_headers + '\n' +
signed_headers + '\n'
146 canonical_request +=
hashlib.sha256(self._construct_query(params).encode('utf-8') if verb
== 'POST' else '').hexdigest()
boto's code is following.
boto/auth.py
283 class HmacAuthV4Handler(AuthHandler, HmacKeys):
393 def canonical_request(self, http_request):
394 cr = [http_request.method.upper()]
395 cr.append(self.canonical_uri(http_request))
396 cr.append(self.canonical_query_string(http_request))
397 headers_to_sign = self.headers_to_sign(http_request)
398 cr.append(self.canonical_headers(headers_to_sign) + '\n')
399 cr.append(self.signed_headers(headers_to_sign))
400 cr.append(self.payload(http_request))
401 return '\n'.join(cr)
337 def canonical_query_string(self, http_request):
338 # POST requests pass parameters in through the
339 # http_request.body field.
340 if http_request.method == 'POST':
341 return ""
342 l = []
343 for param in sorted(http_request.params):
344 value =
boto.utils.get_utf8_value(http_request.params[param])
345 l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),
346 urllib.quote(value, safe='-_.~')))
347 return '&'.join(l)
* Server side
heat-api-cfn queries to keystone in order to check request authorization.
keystone uses keystoneclient to check EC2 format request signature.
In here, **keystoneclient uses (non-empty) query string as
CanonicalQueryString, even
though request is POST.**
And create a digest hash of the CanonicalRequest for signature
calculation.
keystoneclient's code is following.
keystoneclient/contrib/ec2/utils.py
28 class Ec2Signer(object):
154 def _calc_signature_4(self, params, verb, server_string, path,
headers,
155 body_hash):
156 """Generate AWS signature version 4 string."""
235 # Create canonical request:
236 # http://docs.aws.amazon.com/general/latest/gr/
237 # sigv4-create-canonical-request.html
238 # Get parameters and headers in expected string format
239 cr = "\n".join((verb.upper(), path,
240 self._canonical_qs(params),
241 canonical_header_str(),
242 auth_param('SignedHeaders'),
243 body_hash))
125 @staticmethod
126 def _canonical_qs(params):
127 """Construct a sorted, correctly encoded query string as
required for
128 _calc_signature_2 and _calc_signature_4.
129 """
130 keys = list(params)
131 keys.sort()
132 pairs = []
133 for key in keys:
134 val = Ec2Signer._get_utf8_value(params[key])
135 val = urllib.parse.quote(val, safe='-_~')
136 pairs.append(urllib.parse.quote(key, safe='') + '=' + val)
137 qs = '&'.join(pairs)
138 return qs
So it should be different from boto(client side) to
keystoneclient(server side),
cfn-push-stats always fails with error log '403 SignatureDoesNotMatch'
in such reason.
I wrote a patch to resolve how to treat CanonicalQueryString mismatch,
My patch honored AWS original tool and boto, so if request is POST,
'CanonicalQueryString' is regarded as a empty string.
With my patch, Heat instance HA works fine.
This bug affects Heat and Keystone, but patch is only needed in
python-keystoneclient.
So I will report to python-keystoneclient launchpad and submit a patch
to Gerrit.
Please confirm it.
----
My environment is RDO Icehouse/CentOS6.5, and package versions is
following.
* Client side
cloud-init-0.7.4-2.el6.noarch
heat-cfntools-1.2.6-2.el6.noarch
python-boto-2.27.0-1.el6.noarch
* Server side
python-keystoneclient-0.9.0-1.el6.noarch
python-keystone-2014.1.1-1.el6.noarch
openstack-keystone-2014.1.1-1.el6.noarch
----
References
[1]
http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Thanks,
Yukinori Sagara
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
