On 11/02/2018 03:22 PM, Eric Fried wrote:
Based on a (long) discussion yesterday  I have put up a patch 
whereby you can set [compute]resource_provider_association_refresh to
zero and the resource tracker will never* refresh the report client's
provider cache. Philosophically, we're removing the "healing" aspect of
the resource tracker's periodic and trusting that placement won't
diverge from whatever's in our cache. (If it does, it's because the op
hit the CLI, in which case they should SIGHUP - see below.)
- When we initially create the compute node record and bootstrap its
- When the virt driver's update_provider_tree makes a change,
update_from_provider_tree reflects them in the cache as well as pushing
them back to placement.
- If update_from_provider_tree fails, the cache is cleared and gets
rebuilt on the next periodic.
- If you send SIGHUP to the compute process, the cache is cleared.
This should dramatically reduce the number of calls to placement from
the compute service. Like, to nearly zero, unless something is actually
Can I get some initial feedback as to whether this is worth polishing up
into something real? (It will probably need a bp/spec if so.)
In the Queens release, our friends at CERN noticed a serious spike in
the number of requests to placement from compute nodes, even in a
stable-state cloud. Given that we were in the process of adding a ton of
infrastructure to support sharing and nested providers, this was not
unexpected. Roughly, what was previously:
became more like:
# In Queens/Rocky, this would still just return the compute RP
# In Queens/Rocky, this would return nothing
for each provider returned above: # i.e. just one in Q/R
In a cloud the size of CERN's, the load wasn't acceptable. But at the
time, CERN worked around the problem by disabling refreshing entirely.
(The fact that this seems to have worked for them is an encouraging sign
for the proposed code change.)
We're not actually making use of most of that information, but it sets
the stage for things that we're working on in Stein and beyond, like
multiple VGPU types, bandwidth resource providers, accelerators, NUMA,
etc., so removing/reducing the amount of information we look at isn't
really an option strategically.
I support your idea of getting rid of the periodic refresh of the cache
in the scheduler report client. Much of that was added in order to
emulate the original way the resource tracker worked.
Most of the behaviour in the original resource tracker (and some of the
code still in there for dealing with (surprise!) PCI passthrough devices
and NUMA topology) was due to doing allocations on the compute node (the
whole claims stuff). We needed to always be syncing the state of the
compute_nodes and pci_devices table in the cell database with whatever
usage information was being created/modified on the compute nodes .
All of the "healing" code that's in the resource tracker was basically
to deal with "soft delete", migrations that didn't complete or work
properly, and, again, to handle allocations becoming out-of-sync because
the compute nodes were responsible for allocating (as opposed to the
current situation we have where the placement service -- via the
scheduler's call to claim_resources() -- is responsible for allocating
Now that we have generation markers protecting both providers and
consumers, we can rely on those generations to signal to the scheduler
report client that it needs to pull fresh information about a provider
or consumer. So, there's really no need to automatically and blindly
refresh any more.
 We always need to be syncing those tables because those tables,
unlike the placement database's data modeling, couple both inventory AND
usage in the same table structure...
 again, except for PCI devices and NUMA topology, because of the
tight coupling in place with the different resource trackers those types
of resources use...
OpenStack Development Mailing List (not for usage questions)