Reviewed: https://review.opendev.org/c/openstack/nova/+/804968 Committed: https://opendev.org/openstack/nova/commit/d31b26e09d469748f2ae2cc1e5f6f57401834adc Submitter: "Zuul (22348)" Branch: master
commit d31b26e09d469748f2ae2cc1e5f6f57401834adc Author: Fabian Wiesel <[email protected]> Date: Tue Aug 17 16:36:06 2021 +0200 VMWare: Use WithRetrieval to get all results In various places, own version of iterating over the results are implemented, sometimes even faulty. The following functions where only getting up to vmware.maximum_objects objects (100 by default) vm_util.get_all_cluster_mors, vm_util.get_stats_from_cluster. Previously, the results were fetched in batches of up to vmware.maximum_objects items. Using WithRetrieval yields an iterator to the results, which pages transparently to the next request. Consumers of the output of the results where changed to work on an iterator, where easily possible. Replaced the quadratic algorithm in `ds_util._filter_datastores_matching_storage_policy` with one of O(n log(n)) runtime Closes-Bug: #1940399 Change-Id: I8283c3e76c595cb32527d1b8745933d044e22734 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1940399 Title: vmware driver fails when setting vmware.maximum_objects to small value Status in OpenStack Compute (nova): Fix Released Bug description: Description =========== There are two functions in the code, which call (indirectly) RetrievePropertyEx with at most vmware.maximum_objects per result, but do not iterate over the results if there are more, nor does it By default the value is 100, which is usually sufficient, but if someone should set the value lower, it will cause the following problems. Both are in the module nova.virt.vmwarapi.vm_util 1. get_all_cluster_mors The function is called to find the cluster reference for named cluster. If there are more clusters than maximum_objects, then there is a chance that the cluster won't be found by the driver despite being configured in the VCenter. 2. get_stats_from_cluster The function gets the statistics for all the hosts in a cluster. Since VSphere 7.0U1, the limit is 96 hosts per cluster, and quite likely the limit will increase past the default of 100, leaving the stats inaccurate. On top of it, the function does not call `cancel_retrieval` causing a leak in the VCenter. Steps to reproduce ================== Take any VCenter version with more than one cluster and more than one host (I would suggest three or more), and any release of Nova configured to run against the VCenter. I am not 100% according what the order is by which the clusters are returned, possibly in the order they have been created, or alphabetically. I would suggest to create first a cluster-a, and then a cluster-b, and add the hosts to cluster-b. That reflects how our clusters are created (chronologically alphabetically sorted). * Configure additionally [vmware] maximum_objects=1 * Try to start nova-compute Expected result =============== The nova-compute service would start up and get the stats from all the hosts in the configured cluster. Actual result ============= nova-compute fails to start with the error message: > The specified cluster '<clustername>' was not found in vCenter If you have to clusters, and three hosts, then increasing maximum_objects to two will get you around that failure, and will trigger the second problem. You can verify that by checking the resources of the nova-compute node, which will report only the resources (CPUs,RAM...) of two of the ESXi-hosts in the cluster. Environment =========== 1. Exact version of OpenStack you are running. 370830e944 Merge "libvirt: Enable 'vmcoreinfo' feature by default" As far as I can see, all Nova releases are affected by this behavior. 2. Which hypervisor did you use? VMware VSphere What's the version of that? 7.0.1-17327586 3. Which networking type did you use? Neutron with NSX-T (https://github.com/sapcc/networking-nsx-t) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1940399/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
