Public bug reported: A customer is facing an issue where the watcher-decision-engine service crashes when creating an audit plan with the Audit type set to CONTINUOUS. Below are the steps to reproduce the issue:
Environment Details: 1. Deploy Openstack Yoga on Jammy with Watcher and Gnocchi as watcher's storage backend 2. Create an audit openstack optimize audit create --name workload_stabilization_test_1 -s workload_stabilization -g workload_balancing --audit_type CONTINUOUS --interval 60 --auto-trigger 3. Check the audit state openstack optimize audit list Observe it says "CONTINUOUS ONGOING" 4. Create a second audit openstack optimize audit create --name workload_stabilization_test_2 -s workload_stabilization -g workload_balancing --audit_type CONTINUOUS --interval 60 --auto-trigger 5. Check the audit state openstack optimize audit list Observe the second audit is stuck in "CONTINUOUS PENDING" 6. Check watcher's status and observe that it crashed with the following traceback systemctl status watcher-decision-engine.service Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: self.run() Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: File "/usr/lib/python3.10/threading.py", line 953, in run Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: self._target(*self._args, **self._kwargs) Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: File "/usr/lib/python3/dist-packages/apscheduler/schedulers/blocking.py", line 32, in _main_loop Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: wait_seconds = self._process_jobs() Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: File "/usr/lib/python3/dist-packages/apscheduler/schedulers/base.py", line 1006, in _process_jobs Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: jobstore_next_run_time = jobstore.get_next_run_time() Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: File "/usr/lib/python3/dist-packages/apscheduler/jobstores/sqlalchemy.py", line 84, in get_next_run_time Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: return utc_timestamp_to_datetime(float(next_run_time)) Nov 27 19:53:54 juju-2752e1-86-lxd-27 watcher-decision-engine[965896]: TypeError: float() argument must be a string or a real number, not 'NoneType' This was fixed upstream in 2024.2 at https://opendev.org/openstack/watcher/commit/d6f169197efc5b4f6c8a2e6bc38177b0641ca05c which properly addresses the type conversion and https://opendev.org/openstack/watcher/commit/fbb290b2238e9e72054892e9ae6108a8907f47d7 which adjusts the unit tests to accommodate this fix. ** Affects: cloud-archive Importance: Undecided Status: Fix Released ** Affects: cloud-archive/antelope Importance: Undecided Status: New ** Affects: cloud-archive/bobcat Importance: Undecided Status: New ** Affects: cloud-archive/caracal Importance: Undecided Status: New ** Affects: cloud-archive/dalmation Importance: Undecided Status: Fix Released ** Affects: cloud-archive/epoxy Importance: Undecided Status: Fix Released ** Affects: cloud-archive/yoga Importance: Undecided Assignee: Bryan Fraschetti (bryanfraschetti) Status: New ** Affects: cloud-archive/zed Importance: Undecided Status: New ** Affects: watcher (Ubuntu) Importance: Undecided Status: Fix Released ** Affects: watcher (Ubuntu Focal) Importance: Undecided Status: New ** Affects: watcher (Ubuntu Jammy) Importance: Undecided Assignee: Bryan Fraschetti (bryanfraschetti) Status: New ** Affects: watcher (Ubuntu Noble) Importance: Undecided Status: New ** Affects: watcher (Ubuntu Oracular) Importance: Undecided Status: Fix Released ** Affects: watcher (Ubuntu Plucky) Importance: Undecided Status: Fix Released ** Also affects: watcher (Ubuntu Plucky) Importance: Undecided Assignee: Bryan Fraschetti (bryanfraschetti) Status: New ** Also affects: watcher (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: watcher (Ubuntu Oracular) Importance: Undecided Status: New ** Also affects: watcher (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: watcher (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: watcher (Ubuntu Oracular) Status: New => Fix Released ** Changed in: watcher (Ubuntu Plucky) Status: New => Fix Released ** Also affects: watcher Importance: Undecided Status: New ** No longer affects: watcher ** Summary changed: - Watcher crashes on creation of multiple audits and gets stuck in PENDING + [SRU] Watcher crashes on creation of multiple audits and gets stuck in PENDING ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/yoga Importance: Undecided Status: New ** Also affects: cloud-archive/epoxy Importance: Undecided Status: New ** Also affects: cloud-archive/zed Importance: Undecided Status: New ** Also affects: cloud-archive/bobcat Importance: Undecided Status: New ** Also affects: cloud-archive/dalmation Importance: Undecided Status: New ** Also affects: cloud-archive/antelope Importance: Undecided Status: New ** Also affects: cloud-archive/caracal Importance: Undecided Status: New ** Changed in: cloud-archive/dalmation Status: New => Fix Released ** Changed in: cloud-archive/epoxy Status: New => Fix Released ** Changed in: watcher (Ubuntu Jammy) Assignee: (unassigned) => Bryan Fraschetti (bryanfraschetti) ** Changed in: watcher (Ubuntu Plucky) Assignee: Bryan Fraschetti (bryanfraschetti) => (unassigned) ** Changed in: cloud-archive/yoga Assignee: (unassigned) => Bryan Fraschetti (bryanfraschetti) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2091947 Title: [SRU] Watcher crashes on creation of multiple audits and gets stuck in PENDING To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2091947/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
