Public bug reported:
By default, 1000 greenthreads can be used to synchronize the power state
of instances running on a compute node.
In the Ironic context, this means 1000 simultaneous HTTP requests can be
initiated to the Ironic API. Some of those requests can fail with the
following error:
ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic
sync_power_state task had an error while processing an instance.
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most
recent call last):
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6083, in _sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
query_driver_power_state_and_sync()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py",
line 252, in inner
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return f(*args,
**kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6080, in query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
self._query_driver_power_state_and_sync(context, db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6110, in _query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] vm_instance =
self.driver.get_info(db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
557, in get_info
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] node =
_validate_instance_and_node(self.ironicclient, instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
126, in _validate_instance_and_node
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
ironicclient.call("node.get_by_instance_uuid", instance.uuid)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py",
line 122, in call
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
self._multi_getattr(client, method)(*args, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line
151, in get_by_instance_uuid
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] nodes =
self._list(self._path(path), 'nodes')
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line
119, in _list
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body =
self.api.json_request('GET', url)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
351, in json_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body_iter =
self._http_request(url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
160, in wrapper
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return func(self,
url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
296, in _http_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp =
conn.getresponse()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 1051, in getresponse
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] response.begin()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 415, in begin
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] version, status,
reason = self._read_status()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 379, in _read_status
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] raise
BadStatusLine(line)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
There should be a way to limit the number of simultaneous requests made
against the hypervisor to synchronize the instance power states so it's
not overwhelm.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621987
Title:
Power state synchronization in compute is too agressive
Status in OpenStack Compute (nova):
New
Bug description:
By default, 1000 greenthreads can be used to synchronize the power
state of instances running on a compute node.
In the Ironic context, this means 1000 simultaneous HTTP requests can
be initiated to the Ironic API. Some of those requests can fail with
the following error:
ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic
sync_power_state task had an error while processing an instance.
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most
recent call last):
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6083, in _sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
query_driver_power_state_and_sync()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py",
line 252, in inner
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return f(*args,
**kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6080, in query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
self._query_driver_power_state_and_sync(context, db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6110, in _query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] vm_instance =
self.driver.get_info(db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
557, in get_info
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] node =
_validate_instance_and_node(self.ironicclient, instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
126, in _validate_instance_and_node
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
ironicclient.call("node.get_by_instance_uuid", instance.uuid)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py",
line 122, in call
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
self._multi_getattr(client, method)(*args, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line
151, in get_by_instance_uuid
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] nodes =
self._list(self._path(path), 'nodes')
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line
119, in _list
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body =
self.api.json_request('GET', url)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
351, in json_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body_iter
= self._http_request(url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
160, in wrapper
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
func(self, url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
296, in _http_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp =
conn.getresponse()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 1051, in getresponse
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] response.begin()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 415, in begin
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] version,
status, reason = self._read_status()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 379, in _read_status
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] raise
BadStatusLine(line)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
There should be a way to limit the number of simultaneous requests
made against the hypervisor to synchronize the instance power states
so it's not overwhelm.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621987/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
