Public bug reported: This is split out from bug 1807044 where nova-api sometimes takes more than 60 seconds to start on slow CI nodes which causes devstack to timeout and fail.
Specifically, every nova.compute.api.API constructs a SchedulerReportClient, which grabs an in-memory lock per API worker during init: Dec 05 20:14:27.694593 ubuntu-xenial-ovh-bhs1-0000959981 [email protected][23459]: DEBUG oslo_concurrency.lockutils [None req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock "placement_client" released by "nova.scheduler.client.report._create_client" :: held 0.006s {{(pid=23462) inner /usr/local/lib/python2.7/dist- packages/oslo_concurrency/lockutils.py:339}} We could probably be smarter about either making that a singleton in the API or only init on first access since most of the API extensions aren't going to even use that SchedulerReportClient. There are at least 30 extensions in nova-api that create an API class, and there are 2 workers in devstack jobs, and each API class constructs that report client which has a lock during init, so it will have a snowball effect on startup. Furthermore, with this change: https://review.openstack.org/#/c/615641/ The NOTE in the API is no longer true: https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/compute/api.py#L256 So the API likely just needs to add it's own lazy-load behavior for that client. ** Affects: nova Importance: Medium Assignee: Matt Riedemann (mriedem) Status: Triaged ** Tags: api performance ** Changed in: nova Assignee: (unassigned) => Matt Riedemann (mriedem) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1807219 Title: SchedulerReporClient init slows down nova-api startup Status in OpenStack Compute (nova): Triaged Bug description: This is split out from bug 1807044 where nova-api sometimes takes more than 60 seconds to start on slow CI nodes which causes devstack to timeout and fail. Specifically, every nova.compute.api.API constructs a SchedulerReportClient, which grabs an in-memory lock per API worker during init: Dec 05 20:14:27.694593 ubuntu-xenial-ovh-bhs1-0000959981 [email protected][23459]: DEBUG oslo_concurrency.lockutils [None req-dfdfad07-2ff4-43ed-9f67-2acd59687e0c None None] Lock "placement_client" released by "nova.scheduler.client.report._create_client" :: held 0.006s {{(pid=23462) inner /usr/local/lib/python2.7/dist- packages/oslo_concurrency/lockutils.py:339}} We could probably be smarter about either making that a singleton in the API or only init on first access since most of the API extensions aren't going to even use that SchedulerReportClient. There are at least 30 extensions in nova-api that create an API class, and there are 2 workers in devstack jobs, and each API class constructs that report client which has a lock during init, so it will have a snowball effect on startup. Furthermore, with this change: https://review.openstack.org/#/c/615641/ The NOTE in the API is no longer true: https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/compute/api.py#L256 So the API likely just needs to add it's own lazy-load behavior for that client. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1807219/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
