Hi
We are facing few issues with keystone access in our setups.
We have a setup which has three VMs. VM1 and VM3 are on same compute host
whereas VM2 is on a different compute host.
- VM1 (mysite-ceilometer-6): Running Mitaka Ceilometer service + Keystone (v2)
service
- VM2 (mysite-ceilometer-7): Running Ceilometer client
- VM3 (mysite-ceilometer-8): Running Ceilometer client
I have the keystone endpoints setup on VM1 as shown below: 'keystone
endpoint-list'
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
| id | region | publicurl |
internalurl | adminurl |
service_id |
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
| 7c56f9e61 | RegionOne | http://mysite-ceilometer-6:5000/v2.0 |
http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:35357/v2.0 |
4d871 |
| adc3ac02f | RegionOne | http://mysite-ceilometer-6:8777 |
http://mysite-ceilometer-6:8777 | http://mysite-ceilometer-6:8777 |
50fb869 |
+----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+
I have set the following OS variables in both VM2 and VM3:
OS_USERNAME="admin"
OS_PASSWORD="password"
OS_TENANT_NAME="admin"
OS_AUTH_URL="http://mysite-ceilometer-6:5000/v2.0";
The interesting observation I have made is that I am able to run the ceilometer
or keystone commands successfully only from VM3 (which is on the same compute
host as VM1) but not from VM2. Attached some of the logs I have collected
during this observation. Can any of you help us with identifying what is going
wrong here in this setup? Or any pointers on debugging why keystone/tokens REST
API is not responding in some scenarios...? Thanks for ur help.
The output from VM3 (mysite-ceilometer-8):
---------------------------------
root@69861113613c:/usr/local/share# ceilometer -d meter-list
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H
"Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive:
timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date:
Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id:
req-da4329a5-0374-48de-9a07-229636042f64 Content-Type: application/json
X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z",
"media-types": [{"base": "application/json", "type":
"application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links":
[{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href":
"http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}}
DEBUG (v2) Making authentication request to
http://mysite-ceilometer-6:5000/v2.0/tokens
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H
"Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive:
timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date:
Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id:
req-491da351-24c1-4b4d-9558-b6d66a280c35 Content-Type: application/json
X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z",
"media-types": [{"base": "application/json", "type":
"application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links":
[{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href":
"http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}}
DEBUG (v2) Making authentication request to
http://mysite-ceilometer-6:5000/v2.0/tokens
DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293
DEBUG (client) REQ: curl -g -i -X 'GET'
'http://mysite-ceilometer-6:8777/v2/meters' -H 'User-Agent:
ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}a9....'
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2/meters HTTP/1.1" 200 2
DEBUG (client) RESP: [200] {'Date': 'Wed, 23 Nov 2016 22:49:29 GMT',
'Content-Length': '2', 'Content-Type': 'application/json; charset=UTF-8',
'x-openstack-request-id': 'req-6bd80a0c-db08-4907-8b6b-4d477a6d7514', 'Server':
'Werkzeug/0.9.4 Python/2.7.6'}
DEBUG (client) RESP BODY: []
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+
The output from VM2 (mysite-ceilometer-7):
---------------------------------
root@cadf877c4cdb:/usr/local/share# ceilometer -d meter-list
DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 -H
"Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6
DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345
DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token Keep-Alive:
timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date:
Wed, 23 Nov 2016 22:50:12 GMT x-openstack-request-id:
req-f11122bc-bc3c-4521-81f3-d140bfce25b3 Content-Type: application/json
X-Distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z",
"media-types": [{"base": "application/json", "type":
"application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links":
[{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href":
"http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}}
DEBUG (v2) Making authentication request to
http://mysite-ceilometer-6:5000/v2.0/tokens
<<<<<Hangs here until time out>>>
Traceback (most recent call last):
File "/usr/local/bin/ceilometer", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line
292, in main
CeilometerShell().main(args)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", line
238, in main
client = ceiloclient.get_client(api_version, **client_kwargs)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py",
line 366, in get_client
return Client(version, endpoint, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py",
line 313, in Client
return client_class(*args, **client_kwargs)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py",
line 67, in __init__
self.alarm_client = self._get_alarm_client(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py",
line 122, in _get_alarm_client
c.get("/")
File
"/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py",
line 355, in get
return self.client_request("GET", url, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py",
line 345, in client_request
self, method, url, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py",
line 244, in client_request
self.authenticate()
File
"/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py",
line 315, in authenticate
self.auth_plugin.authenticate(self)
File
"/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/auth.py",
line 203, in authenticate
self._do_authenticate(http_client)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py",
line 192, in _do_authenticate
endpoint = _get_endpoint(ks_session, **ks_kwargs)
File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py",
line 162, in _get_endpoint
region_name=kwargs.get('region_name'))
File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line
664, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File
"/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py",
line 324, in get_endpoint
service_catalog = self.get_access(session).service_catalog
File
"/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py",
line 249, in get_access
self.auth_ref = self.get_auth_ref(session)
File
"/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py",
line 88, in get_auth_ref
authenticated=False, log=False)
File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line
505, in post
return self.request(url, 'POST', **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line
337, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line
390, in request
resp = send(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line
434, in _send_request
raise exceptions.ConnectionRefused(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish
connection to http://mysite-ceilometer-6:5000/v2.0/tokens
The output from VM1:
---------------------------------
/mitaka-v2$ sudo tail -f /var/log/keystone/keystone-wsgi-public.log
<<<Successful case>>>>
2016-11-23 22:49:28.612 6830 INFO keystone.common.wsgi
[req-da4329a5-0374-48de-9a07-229636042f64 - - - - -] GET
http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:49:28.713 6826 INFO keystone.common.wsgi
[req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] POST
http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:49:28.831 6826 INFO keystone.token.providers.fernet.utils
[req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] Loaded 2 encryption keys
(max_active_keys=3) from: /etc/keystone/fernet-keys/
2016-11-23 22:49:28.865 6827 INFO keystone.common.wsgi
[req-491da351-24c1-4b4d-9558-b6d66a280c35 - - - - -] GET
http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:49:28.895 6828 INFO keystone.common.wsgi
[req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] POST
http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:49:29.015 6828 INFO keystone.token.providers.fernet.utils
[req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] Loaded 2 encryption keys
(max_active_keys=3) from: /etc/keystone/fernet-keys/
<<<Unsuccessful case>>>>
2016-11-23 22:50:12.331 6829 INFO keystone.common.wsgi
[req-f11122bc-bc3c-4521-81f3-d140bfce25b3 - - - - -] GET
http://mysite-ceilometer-6:5000/v2.0/
2016-11-23 22:50:12.374 6830 INFO keystone.common.wsgi
[req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] POST
http://mysite-ceilometer-6:5000/v2.0/tokens
2016-11-23 22:50:12.481 6830 INFO keystone.token.providers.fernet.utils
[req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] Loaded 2 encryption keys
(max_active_keys=3) from: /etc/keystone/fernet-keys/
Thanks
Srikanth
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
