Public bug reported:

After Train migration we separate placement on another server checking
the endpoint is correct

but when i perform a resize of instance get a error

keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish
connection to
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded
with url:

Checking my endpoint URL on keystone show 172.19.1.26 as Nova API on
port 8774

the problem is nova is mixing Nova IP with placement port the correct
address  should go is 172.19.1.22:8778

nova         | compute      | True    | public    | http://172.19.1.26:8774
placement    | placement    | True    | public    | http://172.19.1.22:8778

before they was together in stein release

as Workaround i use haproxy pointing to the placement ip and port

i add the full stack trace
Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in 
decorated_function return function(self, context, *args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4727, in 
prep_resize filter_properties, host_list) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4802, in 
_reschedule_resize_or_reraise six.reraise(*exc_info) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4710, in 
prep_resize clean_shutdown) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4637, in 
_prep_resize limits=limits) as claim: File 
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in 
inner return f(*args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 255, in 
resize_claim limits=limits) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 378, in 
_move_claim self._update(elevated, cn) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1237, 
in _update self._update_to_placement(context, compute_node, startup) File 
"/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return 
Retrying(*dargs, **dkw).call(f, *args, **kw) File 
"/usr/lib/python3/dist-packages/retrying.py", line 206, in call return 
attempt.get(self._wrap_exception) File 
"/usr/lib/python3/dist-packages/retrying.py", line 247, in get 
six.reraise(self.value[0], self.value[1], self.value[2]) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = 
Attempt(fn(*args, **kwargs), attempt_number, False) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1151, 
in _update_to_placement context, compute_node.uuid, 
name=compute_node.hypervisor_hostname) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 858, in 
get_provider_tree_and_ensure_root parent_provider_uuid=parent_provider_uuid) 
File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 
640, in _ensure_resource_provider rps_to_refresh = 
self.get_providers_in_tree(context, uuid) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 486, in 
get_providers_in_tree global_request_id=context.global_id) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 237, in 
get global_request_id=global_request_id) File 
"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get 
return self.request(url, 'GET', **kwargs) File 
"/usr/lib/python3/dist-packages/openstack/proxy.py", line 98, in request 
**kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 
248, in request return self.session.request(url, method, **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 888, in request 
resp = send(**kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1018, in 
_send_request **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',))

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  After Train migration we separate placement on another server checking
  the endpoint is correct
  
  but when i perform a resize of instance get a error
  
  keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish
  connection to
  
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
  HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded
  with url:
  
  Checking my endpoint URL on keystone show 172.19.1.26 as Nova API on
  port 8774
  
+ the problem is nova is mixing Nova IP with placement port the correct
+ address  should go is 172.19.1.22:8778
+ 
  nova         | compute      | True    | public    | http://172.19.1.26:8774
  placement    | placement    | True    | public    | http://172.19.1.22:8778
- 
  
  before they was together in stein release
  
  as Workaround i use haproxy pointing to the placement ip and port
  
- 
- 
  i add the full stack trace
  Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in 
decorated_function return function(self, context, *args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4727, in 
prep_resize filter_properties, host_list) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4802, in 
_reschedule_resize_or_reraise six.reraise(*exc_info) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4710, in 
prep_resize clean_shutdown) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4637, in 
_prep_resize limits=limits) as claim: File 
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in 
inner return f(*args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 255, in 
resize_claim limits=limits) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 378, in 
_move_claim self._update(elevated, cn) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1237, 
in _update self._update_to_placement(context, compute_node, startup) File 
"/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return 
Retrying(*dargs, **dkw).call(f, *args, **kw) File 
"/usr/lib/python3/dist-packages/retrying.py", line 206, in call return 
attempt.get(self._wrap_exception) File 
"/usr/lib/python3/dist-packages/retrying.py", line 247, in get 
six.reraise(self.value[0], self.value[1], self.value[2]) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = 
Attempt(fn(*args, **kwargs), attempt_number, False) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1151, 
in _update_to_placement context, compute_node.uuid, 
name=compute_node.hypervisor_hostname) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 858, in 
get_provider_tree_and_ensure_root parent_provider_uuid=parent_provider_uuid) 
File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 
640, in _ensure_resource_provider rps_to_refresh = 
self.get_providers_in_tree(context, uuid) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 486, in 
get_providers_in_tree global_request_id=context.global_id) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 237, in 
get global_request_id=global_request_id) File 
"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get 
return self.request(url, 'GET', **kwargs) File 
"/usr/lib/python3/dist-packages/openstack/proxy.py", line 98, in request 
**kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 
248, in request return self.session.request(url, method, **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 888, in request 
resp = send(**kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1018, in 
_send_request **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',))

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1881704

Title:
  Resize don't work on Train when placement is in another Address

Status in OpenStack Compute (nova):
  New

Bug description:
  After Train migration we separate placement on another server checking
  the endpoint is correct

  but when i perform a resize of instance get a error

  keystoneauth1.exceptions.connection.ConnectFailure: Unable to
  establish connection to
  
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
  HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries
  exceeded with url:

  Checking my endpoint URL on keystone show 172.19.1.26 as Nova API on
  port 8774

  the problem is nova is mixing Nova IP with placement port the correct
  address  should go is 172.19.1.22:8778

  nova         | compute      | True    | public    | http://172.19.1.26:8774
  placement    | placement    | True    | public    | http://172.19.1.22:8778

  before they was together in stein release

  as Workaround i use haproxy pointing to the placement ip and port

  i add the full stack trace
  Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn 
(self.host, self.port), self.timeout, **extra_kw) File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in 
create_connection raise err File 
"/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in 
create_connection sock.connect(sa) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect 
socket_checkerr(fd) File 
"/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in 
socket_checkerr raise socket.error(err, errno.errorcode[err]) 
ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above 
exception, another exception occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in 
urlopen chunked=chunked) File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in 
_make_request conn.request(method, url, **httplib_request_kw) File 
"/usr/lib/python3.6/http/client.py", line 1264, in request 
self._send_request(method, url, body, headers, encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1310, in _send_request 
self.endheaders(body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1259, in endheaders 
self._send_output(message_body, encode_chunked=encode_chunked) File 
"/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) 
File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect 
conn = self._new_conn() File 
"/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn 
self, "Failed to establish a new connection: %s" % e) 
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection 
object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send 
timeout=timeout File 
"/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in 
urlopen _stacktrace=sys.exc_info()[2]) File 
"/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment 
raise MaxRetryError(_pool, url, error or ResponseError(cause)) 
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', 
port=8778): Max retries exceeded with url: 
/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by 
NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in 
_send_request resp = self.session.request(method, url, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request 
resp = self.send(prep, **send_kwargs) File 
"/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = 
adapter.send(request, **kwargs) File 
"/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise 
ConnectionError(e, request=request) requests.exceptions.ConnectionError: 
HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',)) During handling of the above exception, another exception 
occurred: Traceback (most recent call last): File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in 
decorated_function return function(self, context, *args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4727, in 
prep_resize filter_properties, host_list) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4802, in 
_reschedule_resize_or_reraise six.reraise(*exc_info) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4710, in 
prep_resize clean_shutdown) File 
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4637, in 
_prep_resize limits=limits) as claim: File 
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in 
inner return f(*args, **kwargs) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 255, in 
resize_claim limits=limits) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 378, in 
_move_claim self._update(elevated, cn) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1237, 
in _update self._update_to_placement(context, compute_node, startup) File 
"/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return 
Retrying(*dargs, **dkw).call(f, *args, **kw) File 
"/usr/lib/python3/dist-packages/retrying.py", line 206, in call return 
attempt.get(self._wrap_exception) File 
"/usr/lib/python3/dist-packages/retrying.py", line 247, in get 
six.reraise(self.value[0], self.value[1], self.value[2]) File 
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File 
"/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = 
Attempt(fn(*args, **kwargs), attempt_number, False) File 
"/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1151, 
in _update_to_placement context, compute_node.uuid, 
name=compute_node.hypervisor_hostname) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 858, in 
get_provider_tree_and_ensure_root parent_provider_uuid=parent_provider_uuid) 
File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 
640, in _ensure_resource_provider rps_to_refresh = 
self.get_providers_in_tree(context, uuid) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 486, in 
get_providers_in_tree global_request_id=context.global_id) File 
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 237, in 
get global_request_id=global_request_id) File 
"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get 
return self.request(url, 'GET', **kwargs) File 
"/usr/lib/python3/dist-packages/openstack/proxy.py", line 98, in request 
**kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 
248, in request return self.session.request(url, method, **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 888, in request 
resp = send(**kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1018, in 
_send_request **kwargs) File 
"/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in 
_send_request raise exceptions.ConnectFailure(msg) 
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish 
connection to 
http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265:
 HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with 
url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused 
by NewConnectionError('<urllib3.connection.HTTPConnection object at 
0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] 
ECONNREFUSED',))

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1881704/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to