Reviewed: https://review.openstack.org/367746
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=386812e287198cd2d340d273753ef06075f7c05d
Submitter: Jenkins
Branch: master
commit 386812e287198cd2d340d273753ef06075f7c05d
Author: Mathieu Gagné <mga...@iweb.com>
Date: Fri Sep 9 00:35:50 2016 -0400
Add sync_power_state_pool_size option
The sync_power_state_pool_size option allows to set
the number of greenthreads available for use to sync power states.
It can be used to reduce the number of concurrent requests
made to the hypervisor or system with real instance power states
for performance reasons.
Closes-bug: #1621987
Change-Id: I9cf900314d71c44ec51eb190c102ddc60e13a767
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621987
Title:
Power state synchronization in compute is too agressive
Status in OpenStack Compute (nova):
Fix Released
Bug description:
By default, 1000 greenthreads can be used to synchronize the power
state of instances running on a compute node.
In the Ironic context, this means 1000 simultaneous HTTP requests can
be initiated to the Ironic API. Some of those requests can fail with
the following error:
ERROR nova.compute.manager [-] [instance: XXXXXXXXXXXXXX] Periodic
sync_power_state task had an error while processing an instance.
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] Traceback (most
recent call last):
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6083, in _sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
query_driver_power_state_and_sync()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py",
line 252, in inner
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return f(*args,
**kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6080, in query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
self._query_driver_power_state_and_sync(context, db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line
6110, in _query_driver_power_state_and_sync
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] vm_instance =
self.driver.get_info(db_instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
557, in get_info
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] node =
_validate_instance_and_node(self.ironicclient, instance)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line
126, in _validate_instance_and_node
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
ironicclient.call("node.get_by_instance_uuid", instance.uuid)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/nova/virt/ironic/client_wrapper.py",
line 122, in call
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
self._multi_getattr(client, method)(*args, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/v1/node.py", line
151, in get_by_instance_uuid
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] nodes =
self._list(self._path(path), 'nodes')
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/base.py", line
119, in _list
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body =
self.api.json_request('GET', url)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
351, in json_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp, body_iter
= self._http_request(url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
160, in wrapper
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] return
func(self, url, method, **kwargs)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/opt/nova/local/lib/python2.7/site-packages/ironicclient/common/http.py", line
296, in _http_request
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] resp =
conn.getresponse()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 1051, in getresponse
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] response.begin()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 415, in begin
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] version,
status, reason = self._read_status()
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] File
"/usr/lib/python2.7/httplib.py", line 379, in _read_status
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] raise
BadStatusLine(line)
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX] BadStatusLine: ''
ERROR nova.compute.manager [instance: XXXXXXXXXXXXXX]
There should be a way to limit the number of simultaneous requests
made against the hypervisor to synchronize the instance power states
so it's not overwhelm.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621987/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
