On 11/02/2018 03:22 PM, Eric Fried wrote:

Based on a (long) discussion yesterday [1] I have put up a patch [2]
whereby you can set [compute]resource_provider_association_refresh to
zero and the resource tracker will never* refresh the report client's
provider cache. Philosophically, we're removing the "healing" aspect of
the resource tracker's periodic and trusting that placement won't
diverge from whatever's in our cache. (If it does, it's because the op
hit the CLI, in which case they should SIGHUP - see below.)

- When we initially create the compute node record and bootstrap its
resource provider.
- When the virt driver's update_provider_tree makes a change,
update_from_provider_tree reflects them in the cache as well as pushing
them back to placement.
- If update_from_provider_tree fails, the cache is cleared and gets
rebuilt on the next periodic.
- If you send SIGHUP to the compute process, the cache is cleared.

This should dramatically reduce the number of calls to placement from
the compute service. Like, to nearly zero, unless something is actually

Can I get some initial feedback as to whether this is worth polishing up
into something real? (It will probably need a bp/spec if so.)

[2] https://review.openstack.org/#/c/614886/

In the Queens release, our friends at CERN noticed a serious spike in
the number of requests to placement from compute nodes, even in a
stable-state cloud. Given that we were in the process of adding a ton of
infrastructure to support sharing and nested providers, this was not
unexpected. Roughly, what was previously:

      GET /resource_providers/$compute_uuid
      GET /resource_providers/$compute_uuid/inventories

became more like:

      # In Queens/Rocky, this would still just return the compute RP
      GET /resource_providers?in_tree=$compute_uuid
      # In Queens/Rocky, this would return nothing
      GET /resource_providers?member_of=...&required=MISC_SHARES...
      for each provider returned above:  # i.e. just one in Q/R
          GET /resource_providers/$compute_uuid/inventories
          GET /resource_providers/$compute_uuid/traits
          GET /resource_providers/$compute_uuid/aggregates

In a cloud the size of CERN's, the load wasn't acceptable. But at the
time, CERN worked around the problem by disabling refreshing entirely.
(The fact that this seems to have worked for them is an encouraging sign
for the proposed code change.)

We're not actually making use of most of that information, but it sets
the stage for things that we're working on in Stein and beyond, like
multiple VGPU types, bandwidth resource providers, accelerators, NUMA,
etc., so removing/reducing the amount of information we look at isn't
really an option strategically.

I support your idea of getting rid of the periodic refresh of the cache in the scheduler report client. Much of that was added in order to emulate the original way the resource tracker worked.

Most of the behaviour in the original resource tracker (and some of the code still in there for dealing with (surprise!) PCI passthrough devices and NUMA topology) was due to doing allocations on the compute node (the whole claims stuff). We needed to always be syncing the state of the compute_nodes and pci_devices table in the cell database with whatever usage information was being created/modified on the compute nodes [0].

All of the "healing" code that's in the resource tracker was basically to deal with "soft delete", migrations that didn't complete or work properly, and, again, to handle allocations becoming out-of-sync because the compute nodes were responsible for allocating (as opposed to the current situation we have where the placement service -- via the scheduler's call to claim_resources() -- is responsible for allocating resources [1]).

Now that we have generation markers protecting both providers and consumers, we can rely on those generations to signal to the scheduler report client that it needs to pull fresh information about a provider or consumer. So, there's really no need to automatically and blindly refresh any more.


[0] We always need to be syncing those tables because those tables, unlike the placement database's data modeling, couple both inventory AND usage in the same table structure...

[1] again, except for PCI devices and NUMA topology, because of the tight coupling in place with the different resource trackers those types of resources use...

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Reply via email to