kiranchavala commented on issue #12383: URL: https://github.com/apache/cloudstack/issues/12383#issuecomment-3782897080
Reopening this as I believe this issue exists and is cricital for users using the usage service on multiple management servers https://docs.cloudstack.apache.org/en/4.22.0.0/installguide/optional_installation.html#installing-the-usage-server-optional Steps to reproduce the issue 1. Have a CloudStack environment (4.20.2 / 4.22) with 2 mgmt servers (mgmt1 and mgmt2) 2. Have the CloudStack usage service running on both management servers ``` [root@ref-trl-10682-k-Mol8-kiran-chavala-mgmt1 ~]# service cloudstack-usage status Redirecting to /bin/systemctl status cloudstack-usage.service ● cloudstack-usage.service - CloudStack Usage Server Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2026-01-22 04:49:14 UTC; 1h 52min ago Docs: http://www.cloudstack.org/ Process: 11161 ExecStartPre=/usr/share/cloudstack-common/scripts/installer/pre-check.sh (code=exited, status=0/SUCCESS) Main PID: 11184 (java) Tasks: 27 (limit: 31267) Memory: 440.8M CGroup: /system.slice/cloudstack-usage.service └─11184 /usr/bin/java -Dpid=11184 -Xms256m -Xmx2048m --add-opens=java.base/java.lang=ALL-UNNAMED -cp /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/c> Jan 22 04:49:14 ref-trl-10682-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Starting CloudStack Usage Server... Jan 22 04:49:14 ref-trl-10682-k-Mol8-kiran-chavala-mgmt1 systemd[1]: Started CloudStack Usage Server. [root@ref-trl-10682-k-Mol8-kiran-chavala-mgmt2 ~]# service cloudstack-usage status Redirecting to /bin/systemctl status cloudstack-usage.service ● cloudstack-usage.service - CloudStack Usage Server Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2026-01-22 04:49:12 UTC; 1h 52min ago Docs: http://www.cloudstack.org/ Process: 7839 ExecStartPre=/usr/share/cloudstack-common/scripts/installer/pre-check.sh (code=exited, status=0/SUCCESS) Main PID: 7862 (java) Tasks: 26 (limit: 31267) Memory: 509.7M CGroup: /system.slice/cloudstack-usage.service └─7862 /usr/bin/java -Dpid=7862 -Xms256m -Xmx2048m --add-opens=java.base/java.lang=ALL-UNNAMED -cp /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/clo> Jan 22 04:49:12 ref-trl-10682-k-Mol8-kiran-chavala-mgmt2 systemd[1]: Starting CloudStack Usage Server... Jan 22 04:49:12 ref-trl-10682-k-Mol8-kiran-chavala-mgmt2 systemd[1]: Started CloudStack Usage Server. ``` ``` mysql> select * from usage_job; +----+-----------------------------------------------------+-------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ | id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat | +----+-----------------------------------------------------+-------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ | 3 | ref-trl-10682-k-Mol8-kiran-chavala-mgmt1/10.0.34.29 | 25630 | 0 | 0 | 0 | 0 | 0 | NULL | NULL | NULL | 2026-01-22 06:50:01 | +----+-----------------------------------------------------+-------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ 1 row in set (0.00 sec) ``` 3. Change the global setting ``` usage.stats.job.aggregation.range to 5 usage.stats.job.exec.time to an upcoming timestamp 07:00 ``` 4. Restart the cloudstack usage service on both management servers service cloudstack-usage restart Check the usage log on both the management servers , the following message should be present saying the usage service will run at 07:00 ``` 2026-01-22 06:48:46,854 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (main:[]) (logid:) Usage is configured to execute in time zone [UTC], at [07:00], each [5] minutes; the current time in that timezone is [2026-01-22T06:48:46+0000] and the next job is scheduled to execute at [2026-01-22T07:00:00+0000]. During its execution, Usage will aggregate stats according to the time zone [GMT] defined in global setting [usage.aggregation.timezone]. 2026-01-22 06:48:58,062 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (main:[]) (logid:) Usage is configured to execute in time zone [UTC], at [07:00], each [5] minutes; the current time in that timezone is [2026-01-22T06:48:58+0000] and the next job is scheduled to execute at [2026-01-22T07:00:00+0000]. During its execution, Usage will aggregate stats according to the time zone [GMT] defined in global setting [usage.aggregation.timezone]. ``` 4. Deploy a vm or perform some activities on cloudstack so that usage events are generated 5. Stop the cloudstack-usage service on mgmt1 service cloudstack-usage stop usage logs on mgmt1 says it have removed the entry in the database ``` 2026-01-22 06:57:04,335 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (Thread-0:[]) (logid:) exiting Usage Manager 2026-01-22 06:57:04,349 INFO [usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_cb74c9ff] (Thread-0:[]) (logid:) Found 1 opens job, to remove 2026-01-22 06:57:04,349 DEBUG [usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_cb74c9ff] (Thread-0:[]) (logid:) Removing job - id: 3, pid: 25630, job type: 0, scheduled: 0, heartbeat: 2026-01-22T06:57:01.000+0000 ``` Check the db ``` mysql> select * from usage_job; Empty set (0.00 sec) ``` Check the Check the usage status on the UI , it says last successful job 56 yrs ago <img width="1605" height="659" alt="Image" src="https://github.com/user-attachments/assets/e2fda535-c6db-4c11-bad4-269d837f90fb"; /> 6. At 07:00 check the usage ogs on mgmt2 ``` 2026-01-22 07:00:00,001 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (Usage-Job-1:[]) (logid:) starting usage job... 2026-01-22 07:00:00,006 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping... 2026-01-22 07:00:00,006 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_6ced2e17] (Usage-Job-1:[]) (logid:) usage job complete ``` 7. I have also waited for 24 hrs a, and the cross checked if usage service on mgmt2 picks up the job Reference log snippets ``` 2026-01-19 11:57:00,002 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job... 2026-01-19 11:57:00,006 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping... 2026-01-19 11:57:00,006 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complet 2026-01-20 11:52:00,003 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complete 2026-01-20 11:57:00,000 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) starting usage job... 2026-01-20 11:57:00,003 DEBUG [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) Not owner of usage job, skipping... 2026-01-20 11:57:00,003 INFO [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_4ff0bd12] (Usage-Job-1:[]) (logid:) usage job complete ``` Workaround Only when then usage service is started on management server 1 , it starts to process the usage jobs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
