This affects a number of other OpenStack projects in a similar way,
including:
- Neutron
- Cinder
- Glance
More projects may be affected that we are unaware of.
=====================================================
Example traceback from Neutron
=====================================================
2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted
('10.0.2.2', 50029) server
/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
msg = self.format(record)
File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76,
in format
return logging.StreamHandler.format(self, record)
File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
return fmt.format(record)
File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line
297, in format
return logging.Formatter.format(self, record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
msg = msg % self.args
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in
as_text
bytes = self.as_bytes()
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in
as_bytes
url = self.url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in
url
url = self.path_url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in
path_url
bpath_info = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68,
in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in
encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid
start byte
Logged from file catch_errors.py, line 41
2016-09-28 00:16:43.489 1218 INFO neutron.wsgi
[req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016
00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984
=====================================================
Example traceback from Cinder
=====================================================
2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted
('10.0.2.2', 49862) server
/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault
[req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: <type
'exceptions.UnicodeDecodeError'> 'utf8' codec can't decode byte 0xc0 in
position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most
recent call last):
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
req.get_response(self.application)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application,
catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter =
application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp =
self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py",
line 323, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response =
req.get_response(self._app)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault application,
catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault app_iter =
application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in
__call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault response =
self.app(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault resp =
self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 817, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault "url":
request.url})
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault url =
self.path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault bpath_info =
bytes_(self.path_info, self.url_encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
req.encget(key, encattr=encattr)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
val.decode(encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault File
"/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault return
codecs.utf_8_decode(input, errors, True)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid
start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault
2016-09-27 23:50:00.224 INFO eventlet.wsgi.server
[req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Traceback (most recent call
last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in
handle_one_response
result = self.application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in
__call__
return app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line
126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in
send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line
126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in
send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/request_id.py",
line 37, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in
send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 81, in __call__
return self._error(ex, req)
File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 56, in _error
msg_dict = dict(url=req.url, status=status)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in
url
url = self.path_url
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in
path_url
bpath_info = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68,
in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in
encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid
start byte
2016-09-27 23:50:00.228 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-
4f7d-b69c-36c385d0fa10 admin] 10.0.2.2 "POST /v2/..%c0%af/backups
HTTP/1.1" status: 500 len: 139 time: 0.0827849
=====================================================
Example traceback from Glance
=====================================================
2016-09-28 00:11:55.844 32495 INFO eventlet.wsgi.server [-] Traceback (most
recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in
handle_one_response
result = self.application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line
126, in __call__
response = req.get_response(self.application)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in
send
application, catch_exc_info=False)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in
call_application
app_iter = application(self.environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in
__call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in
call_func
return self.func(req, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line
123, in __call__
response = self.process_request(req)
File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 148, in
__call__
return self.func(req, *args, **kw)
File
"/usr/local/lib/python2.7/dist-packages/oslo_middleware/healthcheck/__init__.py",
line 361, in process_request
if req.path != self._path:
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 485, in
path
bpath = bytes_(self.path_info, self.url_encoding)
File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68,
in fget
return req.encget(key, encattr=encattr)
File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in
encget
return val.decode(encoding)
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid
start byte
2016-09-28 00:11:55.854 32495 INFO eventlet.wsgi.server [-] 10.0.2.2 - -
[28/Sep/2016 00:11:55] "GET /v2/images/..%c0%af/tags/..%c0%af HTTP/1.1"
500 139 0.043949
I'm not sure whether right approach here is to file a bug with webob (looks
like there are several that haven't been resolved yet [1] [2]), or to fix this
in each project's respective wsgi error-handling code.
[1] https://github.com/Pylons/webob/issues/115
[2] https://github.com/Pylons/webob/issues/161
** Bug watch added: github.com/Pylons/webob/issues #115
https://github.com/Pylons/webob/issues/115
** Bug watch added: github.com/Pylons/webob/issues #161
https://github.com/Pylons/webob/issues/161
** Summary changed:
- String "..%c0%af" causes 500 errors in multiple locations in Keystone v3
+ String "..%c0%af" causes 500 errors in multiple locations
** Also affects: neutron
Importance: Undecided
Status: New
** Also affects: glance
Importance: Undecided
Status: New
** Also affects: cinder
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1613901
Title:
String "..%c0%af" causes 500 errors in multiple locations
Status in Cinder:
New
Status in Glance:
New
Status in OpenStack Identity (keystone):
Confirmed
Status in neutron:
New
Status in OpenStack Security Advisory:
Won't Fix
Bug description:
While doing some testing on Keystone using Syntribos
(https://github.com/openstack/syntribos), our team (myself, Michael
Dong, Rahul U Nair, Vinay Potluri, Aastha Dixit, and Khanak Nangia)
noticed that we got 500 status codes when the string "..%c0%af" was
inserted in various places in the URL for different types of requests.
Here are some examples:
=========
DELETE /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
Content-Length: 0
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:04:27 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-238fd5a9-be45-41f2-893a-97b513b27af3
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
PATCH /v3/policies/..%c0%af HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
Content-type: application/json
X-Auth-Token: [REDACTED]
Content-Length: 70
{"type": "--serialization-mime-type--", "blob": "--serialized-blob--"}
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:05:36 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-57a41600-02b4-4d2a-b3e9-40f7724d65f2
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
GET /v3/domains/0426ac1e48f642ef9544c2251e07e261/groups/..%c0%af/roles
HTTP/1.1
Host: [REDACTED]:5000
Connection: close
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.11.0
X-Auth-Token: [REDACTED]
HTTP/1.1 500 Internal Server Error
Date: Tue, 16 Aug 2016 22:07:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
X-Distribution: Ubuntu
x-openstack-request-id: req-02313f77-63c6-4aa8-a87e-e3d2a13ad6b7
Content-Length: 143
Connection: close
Content-Type: application/json
{"error": {"message": "An unexpected error prevented the server from
fulfilling your request.", "code": 500, "title": "Internal Server
Error"}}
=========
I've marked this as a security issue as a precaution in case it turns
out that there is a more serious vulnerability underlying these
errors. We have no reason to suspect that there is a greater
vulnerability at this time, but given the many endpoints this seems to
affect, I figured caution was worthwhile since this may be a
framework-wide issue. Feel free to make this public if it is
determined not to be security-impacting.
Here is a (possibly incomplete) list of affected endpoints. Inserting
the string "..%c0%af" in any or all of the spots labeled "HERE" should
yield a 500 error. As you can see, virtually all v3 endpoints exhibit
this behavior.
=========
[GET|PATCH|DELETE] /v3/endpoints/[HERE]
[GET|PATCH] /v3/domains/[HERE]
GET /v3/domains/[HERE]/groups/[HERE]/roles
[HEAD|PUT|DELETE] /v3/domains/[HERE]/groups/[HERE]/roles/[HERE]
GET /v3/domains/[HERE]/users/[HERE]/roles
[HEAD|DELETE] /v3/domains/[HERE]/users/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/groups/[HERE]
[HEAD|PUT|DELETE] /v3/groups[HERE]/users/[HERE]
[POST|DELETE] /v3/keys/[HERE]
[GET|PATCH|DELETE] /v3/policies/[HERE]
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/endpoints/[HERE]
[GET|HEAD] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/policy
[GET|PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[PUT|DELETE] /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
[GET|PUT|DELETE]
/v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/regions/[HERE]
[GET|PATCH|DELETE] /v3/projects/[HERE]
[DELETE|PATCH] /v3/projects/[HERE]/cascade
GET /v3/projects/[HERE]/groups/[HERE]/roles
GET /v3/projects/[HERE]/users/[HERE]/roles
[HEAD|PUT|DELETE] /v3/projects/[HERE]/groups/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/regions/[HERE]
[PATCH|DELETE] /v3/roles/[HERE]
[GET|PATCH|DELETE] /v3/services/[HERE]
[GET|PATCH|DELETE] /v3/users/[HERE]
GET /v3/users/[HERE]/groups
POST /v3/users/[HERE]/password
GET /v3/users/[HERE]/projects
GET
/v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]/roles/[HERE]
[GET|PATCH|DELETE] /v3/OS-OAUTH1/consumers/[HERE]
[GET|DELETE] /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1613901/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
