I faced this problem on the gate job like http://logs.openstack.org/59/426459/5/check/gate-tempest-dsvm-neutron- dvr-ubuntu-xenial/c3d2fac/logs
nova-api could not connect to neutron-server via keystoneauth1 like http://logs.openstack.org/59/426459/5/check/gate-tempest-dsvm-neutron- dvr-ubuntu- xenial/c3d2fac/logs/screen-n-api.txt.gz#_2017-02-10_17_26_20_560 Traceback (most recent call last): File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped return f(*args, **kwargs) File "/opt/stack/new/nova/nova/api/openstack/compute/attach_interfaces.py", line 74, in show port_info = self.network_api.show_port(context, port_id) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1243, in show_port return dict(port=self._show_port(context, port_id)) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1261, in _show_port result = neutron_client.show_port(port_id) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 760, in show_port return self.get(self.port_path % (port), params=_params) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 361, in get headers=headers, params=params) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 338, in retry_request headers=headers, params=params) File "/opt/stack/new/nova/nova/network/neutronv2/api.py", line 99, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 289, in do_request resp, replybody = self.httpclient.do_request(action, method, body=body) File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 314, in do_request return self.request(url, method, **kwargs) File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 302, in request resp = super(SessionClient, self).request(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 142, in request return self.session.request(url, method, **kwargs) File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner return wrapped(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 616, in request resp = send(**kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 690, in _send_request raise exceptions.ConnectFailure(msg) ConnectFailure: Unable to establish connection to https://10.25.148.68:9696/v2.0/ports/d5a00be6-bb0e-4911-b1a9-ffbd25544947.json: ('Connection aborted.', BadStatusLine("''",)) and neutron-server's log doesn't contain any access log about this. ** Changed in: keystone Status: Opinion => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1535640 Title: Sometimes getting unable to establish connection error in Keystone client Status in OpenStack Identity (keystone): Opinion Bug description: In Openstack HA environment, sometimes Kesystone client is throwing the error "unable to establish connection error". This is happening once in fifteen times. The error and stack trace of the error log is given below. Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks 2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624 Stack trace of the error: NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP Traceback (most recent call last): File "/usr/bin/heat", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main HeatShell().main(args) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main args.func(client, args) File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create hc.stacks.create(**fields) File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create data=kwargs, headers=headers) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request raise_exc=False, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request resp = send(**kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request raise exceptions.ConnectionRefused(msg) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks I think we should retry the API connection from keystone client whenever there is a failure of connection to API. When I added code to retry it is working fine. In the existing code Keystoneclient is trying only one time and returning an error if it is not able to connect to any other service API. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1535640/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
