i would probably just try to get a token from vm2 and see if that's even possible. seems unrelated to ceilometerclient.
that said, i'm going to throw in obligatory: "ceilometer api is deprecated and has been unmaintained for over a year. i suggest you switch to another storage solution whether that's gnocchi or another time-series service" On 23/11/16 06:35 PM, Srikanth Vavilapalli wrote: > Hi > > We are facing few issues with keystone access in our setups. > > We have a setup which has three VMs. VM1 and VM3 are on same compute host > whereas VM2 is on a different compute host. > > - VM1 (mysite-ceilometer-6): Running Mitaka Ceilometer service + Keystone > (v2) service > - VM2 (mysite-ceilometer-7): Running Ceilometer client > - VM3 (mysite-ceilometer-8): Running Ceilometer client > > I have the keystone endpoints setup on VM1 as shown below: 'keystone > endpoint-list' > +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+ > | id | region | publicurl | > internalurl | adminurl | > service_id | > +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+ > | 7c56f9e61 | RegionOne | http://mysite-ceilometer-6:5000/v2.0 | > http://mysite-ceilometer-6:5000/v2.0 | http://mysite-ceilometer-6:35357/v2.0 > | 4d871 | > | adc3ac02f | RegionOne | http://mysite-ceilometer-6:8777 | > http://mysite-ceilometer-6:8777 | http://mysite-ceilometer-6:8777 | > 50fb869 | > +----------------------------------+-----------+--------------------------------------+--------------------------------------+---------------------------------------+----------------------------------+ > > I have set the following OS variables in both VM2 and VM3: > OS_USERNAME="admin" > OS_PASSWORD="password" > OS_TENANT_NAME="admin" > OS_AUTH_URL="http://mysite-ceilometer-6:5000/v2.0"; > > The interesting observation I have made is that I am able to run the > ceilometer or keystone commands successfully only from VM3 (which is on the > same compute host as VM1) but not from VM2. Attached some of the logs I have > collected during this observation. Can any of you help us with identifying > what is going wrong here in this setup? Or any pointers on debugging why > keystone/tokens REST API is not responding in some scenarios...? Thanks for > ur help. > > The output from VM3 (mysite-ceilometer-8): > --------------------------------- > root@69861113613c:/usr/local/share# ceilometer -d meter-list > DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 > -H "Accept: application/json" -H "User-Agent: python-keystoneclient" > INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6 > DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345 > DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token > Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: > Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: > req-da4329a5-0374-48de-9a07-229636042f64 Content-Type: application/json > X-Distribution: Ubuntu > RESP BODY: {"version": {"status": "stable", "updated": > "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": > "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": > [{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href": > "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}} > > DEBUG (v2) Making authentication request to > http://mysite-ceilometer-6:5000/v2.0/tokens > DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293 > DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 > -H "Accept: application/json" -H "User-Agent: python-keystoneclient" > INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6 > DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345 > DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token > Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: > Keep-Alive Date: Wed, 23 Nov 2016 22:49:28 GMT x-openstack-request-id: > req-491da351-24c1-4b4d-9558-b6d66a280c35 Content-Type: application/json > X-Distribution: Ubuntu > RESP BODY: {"version": {"status": "stable", "updated": > "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": > "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": > [{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href": > "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}} > > DEBUG (v2) Making authentication request to > http://mysite-ceilometer-6:5000/v2.0/tokens > DEBUG (connectionpool) "POST /v2.0/tokens HTTP/1.1" 200 1293 > DEBUG (client) REQ: curl -g -i -X 'GET' > 'http://mysite-ceilometer-6:8777/v2/meters' -H 'User-Agent: > ceilometerclient.openstack.common.apiclient' -H 'X-Auth-Token: {SHA1}a9....' > INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6 > DEBUG (connectionpool) "GET /v2/meters HTTP/1.1" 200 2 > DEBUG (client) RESP: [200] {'Date': 'Wed, 23 Nov 2016 22:49:29 GMT', > 'Content-Length': '2', 'Content-Type': 'application/json; charset=UTF-8', > 'x-openstack-request-id': 'req-6bd80a0c-db08-4907-8b6b-4d477a6d7514', > 'Server': 'Werkzeug/0.9.4 Python/2.7.6'} > > DEBUG (client) RESP BODY: [] > > +------+------+------+-------------+---------+------------+ > | Name | Type | Unit | Resource ID | User ID | Project ID | > +------+------+------+-------------+---------+------------+ > +------+------+------+-------------+---------+------------+ > > The output from VM2 (mysite-ceilometer-7): > --------------------------------- > > root@cadf877c4cdb:/usr/local/share# ceilometer -d meter-list > DEBUG (session) REQ: curl -g -i -X GET http://mysite-ceilometer-6:5000/v2.0 > -H "Accept: application/json" -H "User-Agent: python-keystoneclient" > INFO (connectionpool) Starting new HTTP connection (1): mysite-ceilometer-6 > DEBUG (connectionpool) "GET /v2.0 HTTP/1.1" 200 345 > DEBUG (session) RESP: [200] Content-Length: 345 Vary: X-Auth-Token > Keep-Alive: timeout=5, max=100 Server: Apache/2.4.7 (Ubuntu) Connection: > Keep-Alive Date: Wed, 23 Nov 2016 22:50:12 GMT x-openstack-request-id: > req-f11122bc-bc3c-4521-81f3-d140bfce25b3 Content-Type: application/json > X-Distribution: Ubuntu > RESP BODY: {"version": {"status": "stable", "updated": > "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": > "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": > [{"href": "http://mysite-ceilometer-6:5000/v2.0/";, "rel": "self"}, {"href": > "http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}} > > DEBUG (v2) Making authentication request to > http://mysite-ceilometer-6:5000/v2.0/tokens > <<<<<Hangs here until time out>>> > Traceback (most recent call last): > File "/usr/local/bin/ceilometer", line 11, in <module> > sys.exit(main()) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", > line 292, in main > CeilometerShell().main(args) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/shell.py", > line 238, in main > client = ceiloclient.get_client(api_version, **client_kwargs) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", > line 366, in get_client > return Client(version, endpoint, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", > line 313, in Client > return client_class(*args, **client_kwargs) > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line > 67, in __init__ > self.alarm_client = self._get_alarm_client(**kwargs) > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line > 122, in _get_alarm_client > c.get("/") > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", > line 355, in get > return self.client_request("GET", url, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", > line 345, in client_request > self, method, url, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", > line 244, in client_request > self.authenticate() > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", > line 315, in authenticate > self.auth_plugin.authenticate(self) > File > "/usr/local/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/auth.py", > line 203, in authenticate > self._do_authenticate(http_client) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", > line 192, in _do_authenticate > endpoint = _get_endpoint(ks_session, **ks_kwargs) > File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", > line 162, in _get_endpoint > region_name=kwargs.get('region_name')) > File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", > line 664, in get_endpoint > return auth.get_endpoint(self, **kwargs) > File > "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", > line 324, in get_endpoint > service_catalog = self.get_access(session).service_catalog > File > "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", > line 249, in get_access > self.auth_ref = self.get_auth_ref(session) > File > "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", > line 88, in get_auth_ref > authenticated=False, log=False) > File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", > line 505, in post > return self.request(url, 'POST', **kwargs) > File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line > 337, in inner > return func(*args, **kwargs) > File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", > line 390, in request > resp = send(**kwargs) > File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", > line 434, in _send_request > raise exceptions.ConnectionRefused(msg) > keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish > connection to http://mysite-ceilometer-6:5000/v2.0/tokens > > > The output from VM1: > --------------------------------- > > /mitaka-v2$ sudo tail -f /var/log/keystone/keystone-wsgi-public.log > > <<<Successful case>>>> > 2016-11-23 22:49:28.612 6830 INFO keystone.common.wsgi > [req-da4329a5-0374-48de-9a07-229636042f64 - - - - -] GET > http://mysite-ceilometer-6:5000/v2.0/ > 2016-11-23 22:49:28.713 6826 INFO keystone.common.wsgi > [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] POST > http://mysite-ceilometer-6:5000/v2.0/tokens > 2016-11-23 22:49:28.831 6826 INFO keystone.token.providers.fernet.utils > [req-96cfc836-e23e-4cd8-a579-da312803d160 - - - - -] Loaded 2 encryption keys > (max_active_keys=3) from: /etc/keystone/fernet-keys/ > 2016-11-23 22:49:28.865 6827 INFO keystone.common.wsgi > [req-491da351-24c1-4b4d-9558-b6d66a280c35 - - - - -] GET > http://mysite-ceilometer-6:5000/v2.0/ > 2016-11-23 22:49:28.895 6828 INFO keystone.common.wsgi > [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] POST > http://mysite-ceilometer-6:5000/v2.0/tokens > 2016-11-23 22:49:29.015 6828 INFO keystone.token.providers.fernet.utils > [req-6a41fb86-fd13-4d92-bcef-78ffc5199336 - - - - -] Loaded 2 encryption keys > (max_active_keys=3) from: /etc/keystone/fernet-keys/ > > <<<Unsuccessful case>>>> > 2016-11-23 22:50:12.331 6829 INFO keystone.common.wsgi > [req-f11122bc-bc3c-4521-81f3-d140bfce25b3 - - - - -] GET > http://mysite-ceilometer-6:5000/v2.0/ > 2016-11-23 22:50:12.374 6830 INFO keystone.common.wsgi > [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] POST > http://mysite-ceilometer-6:5000/v2.0/tokens > 2016-11-23 22:50:12.481 6830 INFO keystone.token.providers.fernet.utils > [req-abaf2a35-6a46-4a81-93a4-cce26f49de80 - - - - -] Loaded 2 encryption keys > (max_active_keys=3) from: /etc/keystone/fernet-keys/ > > > Thanks > Srikanth > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- gord __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
