Re: [openstack-dev] [nova] [placement] compute nodes use of placement
On 7/30/2018 1:55 PM, Jay Pipes wrote: ack. will review shortly. thanks, Chris. For those on the edge of their seats at home, we have merged [1] in Stein and assuming things don't start failing in weird ways after some period of time, we'll probably backport it. OVH is already running with it. [1] https://review.openstack.org/#/c/520024/ -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] compute nodes use of placement
ack. will review shortly. thanks, Chris. On 07/30/2018 02:20 PM, Chris Dent wrote: On Mon, 30 Jul 2018, Jay Pipes wrote: On 07/26/2018 12:15 PM, Chris Dent wrote: The `in_tree` calls happen from the report client method `_get_providers_in_tree` which is called by `_ensure_resource_provider` which can be called from multiple places, but in this case is being called both times from `get_provider_tree_and_ensure_root`, which is also responsible for two of the inventory request. `get_provider_tree_and_ensure_root` is called by `_update` in the resource tracker. `_update` is called by both `_init_compute_node` and `_update_available_resource`. Every single period job iteration. `_init_compute_node` is called from _update_available_resource` itself. That accounts for the overall doubling. Actually, no. What accounts for the overall doubling is the fact that we no longer short-circuit return from _update() when there are no known changes in the node's resources. I think we're basically agreeing on this: I'm describing the current state of affairs, not attempting to describe why it is that way. Your insight helps to explain why. I have a set of change in progress which experiments with what happens if we don't call placement a second time in the _update call: https://review.openstack.org/#/c/587050/ Just to see what might blow up. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] compute nodes use of placement
On Mon, 30 Jul 2018, Jay Pipes wrote: On 07/26/2018 12:15 PM, Chris Dent wrote: The `in_tree` calls happen from the report client method `_get_providers_in_tree` which is called by `_ensure_resource_provider` which can be called from multiple places, but in this case is being called both times from `get_provider_tree_and_ensure_root`, which is also responsible for two of the inventory request. `get_provider_tree_and_ensure_root` is called by `_update` in the resource tracker. `_update` is called by both `_init_compute_node` and `_update_available_resource`. Every single period job iteration. `_init_compute_node` is called from _update_available_resource` itself. That accounts for the overall doubling. Actually, no. What accounts for the overall doubling is the fact that we no longer short-circuit return from _update() when there are no known changes in the node's resources. I think we're basically agreeing on this: I'm describing the current state of affairs, not attempting to describe why it is that way. Your insight helps to explain why. I have a set of change in progress which experiments with what happens if we don't call placement a second time in the _update call: https://review.openstack.org/#/c/587050/ Just to see what might blow up. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] compute nodes use of placement
On 07/26/2018 12:15 PM, Chris Dent wrote: The `in_tree` calls happen from the report client method `_get_providers_in_tree` which is called by `_ensure_resource_provider` which can be called from multiple places, but in this case is being called both times from `get_provider_tree_and_ensure_root`, which is also responsible for two of the inventory request. `get_provider_tree_and_ensure_root` is called by `_update` in the resource tracker. `_update` is called by both `_init_compute_node` and `_update_available_resource`. Every single period job iteration. `_init_compute_node` is called from _update_available_resource` itself. That accounts for the overall doubling. Actually, no. What accounts for the overall doubling is the fact that we no longer short-circuit return from _update() when there are no known changes in the node's resources. We *used* to do a quick check of whether the resource tracker's local cache of resources had been changed, and just exit _update() if no changes were detected. However, this patch modified that so that we *always* call to get inventory, even if the resource tracker noticed no changes in resources: https://github.com/openstack/nova/commit/e2a18a37190e4c7b7697a8811553d331e208182c The reason for that change is because the virt driver was tracking vGPU resources now and those vGPU resources were not tracked by the resource tracker's local cache of resources. Thus, we now always call the virt driver get_inventory() call (which morphed into the virt driver's update_provider_tree() call, but the change to update_provider_tree() didn't actually increase the number of calls to get inventories. It was the patch above that did that. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] [placement] compute nodes use of placement
HTML: https://anticdent.org/novas-use-of-placement.html A year and a half ago I did some analysis on how [nova uses placement](http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html). I've repeated some of that analysis today and here's a brief summary of the results. Note that I don't present this because I'm concerned about load on placement, we've demonstrated that placement scales pretty well. Rather, this analysis indicates that the compute node is doing redundant work which we'd prefer not to do. The compute node can't scale horizontally in the same way placement does. If offloading the work to placement and being redundant is the easiest way to avoid work on the compute node, let's do that, but that doesn't seem to be quite what's happening here. Nova uses placement mainly from two places: * The `nova-compute` nodes report resource provider and inventory to placement and make sure that the placement view of what hardware is present is accurate. * The `nova-scheduler` processes request candidates for placement, and claim resources by writing allocations to placement. There are some additional interactions, mostly associated with migrations or fixing up unusual edge cases. Since those things are rare they are sort of noise in this discussion, so left out. When a basic (where basic means no nested resource providers) compute node starts up it POSTs to create a resource provider and then PUTs to set the inventory. After that a periodic job runs, usually every 60 seconds. In that job we see the following 11 requests: GET /placement/resource_providers?in_tree=82fffbc6-572b-4db0-b044-c47e34b27ec6 GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/inventories GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/aggregates GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/traits GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/inventories GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/allocations GET /placement/resource_providers?in_tree=82fffbc6-572b-4db0-b044-c47e34b27ec6 GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/inventories GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/aggregates GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/traits GET /placement/resource_providers/82fffbc6-572b-4db0-b044-c47e34b27ec6/inventories A year and a half ago it was 5 requests per-cycle, but they were different requests: GET /placement/resource_providers/0e33c6f5-62f3-4522-8f95-39b364aa02b4/aggregates GET /placement/resource_providers/0e33c6f5-62f3-4522-8f95-39b364aa02b4/inventories GET /placement/resource_providers/0e33c6f5-62f3-4522-8f95-39b364aa02b4/allocations GET /placement/resource_providers/0e33c6f5-62f3-4522-8f95-39b364aa02b4/aggregates GET /placement/resource_providers/0e33c6f5-62f3-4522-8f95-39b364aa02b4/inventories The difference comes from two changes: * We no longer confirm allocations on the compute node. * We've now have things called ProviderTrees which are responsible for managing nested providers, aggregates and traits in a unified fashion. It appears, however, that we have some redundancies. We get inventories 4 times; aggregates, providers and traits 2 times, and allocations once. The `in_tree` calls happen from the report client method `_get_providers_in_tree` which is called by `_ensure_resource_provider` which can be called from multiple places, but in this case is being called both times from `get_provider_tree_and_ensure_root`, which is also responsible for two of the inventory request. `get_provider_tree_and_ensure_root` is called by `_update` in the resource tracker. `_update` is called by both `_init_compute_node` and `_update_available_resource`. Every single period job iteration. `_init_compute_node` is called from _update_available_resource` itself. That accounts for the overall doubling. The two calls inventories per group come from the following, in `get_provider_tree_and_ensure_root`: 1. `_ensure_resource_provider` in the report client calls `_refresh_and_get_inventory` for every provider in the tree (the result of the `in_tree` query) 2. Immediately after the the call to `_ensure_resource_provider` every provider in the provider tree (from `self._provider_tree.get_provider_uuids()`) then has a `_refresh_and_get_inventory` call made. In a non-sharing, non-nested scenario (such as a single node devstack, which is where I'm running this analysis) these are the exact same one resource provider. I'm insufficiently aware of what might be in the provider tree in more complex situations to be clear on what could be done to limit redundancy here, but it's a place worth looking. The requests for aggregates and traits happen via `_refresh_associations` in
