** Description changed: [ Impact ] * The watcher releases targeted by this SRU are using a deprecated ceilometer metric, cpu_util, which previously reported cpu utilization as a percentage. This metric was deprecated in Openstack Rocky in favor of "the gnocchi rate calculation equivalent" [1] - essentially meaning that the cpu utilization value should be obtained by performing a calculation with gnocchi's rates. The ceilometer metric cpu_util was - then fully removed in Victoria. + then fully removed in Stein. * Upstream Watcher continued to use cpu_util until the commit at [2] landed on master for 2024.1. Since the ceilometer no longer has a cpu_util metric, polling this metric returns "None". What this means is that all Watcher strategies, particularly those relating to workload balancing and migration of VMs to under-utilized hosts, which rely on - cpu_util are non-functional from Victoria, when the metric was removed, + cpu_util are non-functional from Stein, when the metric was removed, until Caracal. - * This commit correctly performs the cpu calculation as intended using - gnocchi. The calculation is summarized in the next bullet point and - there is an example calculation in the original commit + * This commit consumes the correct metric (cpu) and performs the + utilization calculation as intended using gnocchi's rates. The + calculation is summarized in the next bullet point and there is an + example calculation in the original commit * Gnocchi uses the cumulative cpu time in ns (reported by the ceilometer metric, "cpu") and consumes it as a rate (essentially it computes the difference in cumulative cpu time over the last two sampling intervals) to find the total cpu time during the previous sampling period. Dividing the cpu time in one interval by the duration of the interval multiplied by the number of vcpus provides the cpu utilization as a percentage: cpu_usage = [cpu_time / (period * 10^9 * nvcpus)] * 100%. A sample calculation is provided in the original commit message. * I cherry-picked to stable/2023.2 [3], but the other branches have gone unmaintained [ Test Plan ] * Deploy openstack yoga on jammy with watcher and gnocchi services * Launch a server and take note of it's resource id. Then find the gnocchi cpu metric associated with the instance * Create a watcher audit based on a goal that previously depended on instance cpu utilization (from Watcher's perspective this is called instance_cpu_usage). For example the workload_balance goal [4] depends on instance_cpu_usage Ex. openstack optimize audit create -t CONTINUOUS -i 60 -g workload_balancing -s workload_balance --auto-trigger * Without the patch, the workload_balance strategy does not work. The audit will be created, but it cannot provide any meaningful action plan - since instance_cpu_usage is None in the audits. This can be tested by - stressing the VM such that Watcher should suggest a migration. With the - patch Watcher obtains the correct cpu utilization percentage and the - strategies work as expected. + since instance_cpu_usage is None in the audits. With the patch Watcher + obtains the correct cpu utilization percentage and the strategies work + as expected. * Wait for at least one sampling period to elapse and check /var/log/watcher/watcher-decision-engine.log for entries showing "instance_cpu_usage" - this is the cpu utilization as a percentage. * To verify the percentage with a manual calculation, run gnocchi measure show <metric uuid> --aggregation "rate:mean" and perform the calculation instance_cpu_usage = 100*[<value> / (period * 10^9 * nvcpus) using the cpu time from the corresponding sampling period [ What can go wrong ] * While the patch restores functionality by calculating cpu utilization using gnocchi's rate metric, if gnocchi is misconfigured or the relevant "cpu" metric is missing, the new calculation may not work as anticipated [1] https://docs.openstack.org/releasenotes/ceilometer/rocky.html [2] https://review.opendev.org/c/openstack/watcher/+/898791 [3] https://review.opendev.org/c/openstack/watcher/+/934181 [4] https://docs.openstack.org/watcher/2024.1/strategies/workload_balance.html
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2088620 Title: [SRU] Deprecated usage of cpu_util To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2088620/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
