Reviewed: https://review.openstack.org/587636 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b5b7d86bb04f92d21cf954cd6b3463c9fcc637e6 Submitter: Zuul Branch: master
commit b5b7d86bb04f92d21cf954cd6b3463c9fcc637e6 Author: Matt Riedemann <[email protected]> Date: Tue Jul 31 17:26:47 2018 -0400 Make ResourceTracker.stats node-specific As of change I6827137f35c0cb4f9fc4c6f753d9a035326ed01b in Ocata, the ResourceTracker manages multiple compute nodes via its "compute_nodes" variable, but the "stats" variable was still being shared across all nodes, which leads to leaking stats across nodes in an ironic deployment where a single nova-compute service host is managing multiple ironic instances (nodes). This change makes ResourceTracker.stats node-specific which fixes the ironic leak but also allows us to remove the stats deepcopy while iterating over instances which should improve performance for single-node deployments with potentially a large number of instances, i.e. vCenter. Change-Id: I0b9e5b711878fa47ba90e43c0b41437b57cf8ef6 Closes-Bug: #1784705 Closes-Bug: #1777422 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1784705 Title: ResourceTracker.stats can leak across multiple ironic nodes Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: In Progress Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: In Progress Bug description: A single nova-compute service host can manage multiple ironic nodes, which creates multiple ComputeNode records per compute service host, and ironic instances are 1:1 with each compute node. Before change https://review.openstack.org/#/c/398473/ in Ocata, the ComputeManager would manage multiple ResourceTracker instances, one per compute node (so one per ironic instance managed by that host). As a result of that change, the ComputeManager manages a single ResourceTracker instance, and the ResourceTracker's compute_node entry was changed to a dict, so the RT could manage multiple compute nodes (one per ironic instance). The problem is the ResourceTracker.stats variable was left to be "shared" across all compute nodes being managed by the single RT, which can cause problems in the ResourceTracker._update_usage_from_instance() method which updates the stats and then assigns it to a compute node record, so it could leak/accumulate information about the stats for a different node. The compute node stats are used by the ComputeCapabilitiesFilter in the scheduler so it could be possible for a compute node B to be reporting node capabilities which only apply to node A. This was discovered during code review of this change: https://review.openstack.org/#/c/576099/2/nova/compute/resource_tracker.py@1130 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1784705/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
