Hi. The query results in this: ``` mysql> select * from cloud_usage.usage_job where end_millis = 0; +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ | id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat | +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ | 1 | NULL | 0 | 1 | 0 | 0 | 0 | 0 | NULL | NULL | NULL | 2024-05-01 09:00:15 | +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ 1 row in set (0.00 sec) ```
So I understand correctly this stray entry might be causing the issue and PR #8590 was meant to address it and #10848 would've resolved this automatically? The hostname is cloudstack.wavecom.ee with a load balancer in front on different hostname for public consumption. Lugupidamisega / Best regards, Joann Mõndresku Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee ----- Original Message ----- From: "Fabricio Duarte" <fabricio.dua...@scclouds.com.br> To: "users" <users@cloudstack.apache.org> Sent: Wednesday, May 28, 2025 4:30:37 PM Subject: Re: Problem with CloudStack Usage The DB connection is ok, and the error you witnessed is unrelated. It seems that there are some situations in which an Usage server is unable to takeover unfinished jobs and execute. This issue may be related to PRs [1] and [2]. Could you share the result of the following query so that we can investigate this further? ``` select * from cloud_usage.usage_job where end_millis = 0; ``` Also, what is your hostname? Thank you. [1]: https://github.com/apache/cloudstack/pull/10848 [2]: https://github.com/apache/cloudstack/pull/8896 On 5/28/25 03:49, Joann Mõndresku wrote: > Hi. > > /etc/cloudstack/usage/db.properties is symlinked to > /etc/cloudstack/management/db.properties > > It has these values defined: > ``` > db.usage.username=cloud > db.usage.password=ENC(Encrypted Password) > db.usage.host=localhost > db.usage.driver=jdbc:mysql > db.usage.port=3306 > db.usage.name=cloud_usage > db.usage.uri= > db.usage.maxActive=100 > db.usage.maxIdle=30 > db.usage.maxWait=10000 > db.usage.url.params=serverTimezone=UTC > db.usage.replicas=localhost,localhost > db.usage.autoReconnect=true > db.usage.failOverReadOnly=false > db.usage.reconnectAtTxEnd=true > db.usage.autoReconnectForPools=true > db.usage.secondsBeforeRetrySource=3600 > db.usage.queriesBeforeRetrySource=5000 > db.usage.initialTimeout=3600 > ``` > If I change any of the usage db credentials in that file as a test, it will > cause an error, so it is reading the properties from file seemingly without > issue. > > The only issue in usage.err witnessed: > ``` > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > ``` > > usage.log pastebin from start to ready: https://pastebin.com/Dw6jy164 > > Checking with `cat /var/log/cloudstack/management/management-server.log | > grep "UsageServerMonitorTask"`, every 10min this is shown: > ``` > 2025-05-28 08:41:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) checking health of usage server > 2025-05-28 08:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 08:51:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-d18e3a94]) (logid:4bda154f) checking health of usage server > 2025-05-28 08:51:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-d18e3a94]) (logid:4bda154f) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 09:01:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) checking health of usage server > 2025-05-28 09:01:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 09:11:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-e9020dff]) (logid:88afe31f) checking health of usage server > 2025-05-28 09:11:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-e9020dff]) (logid:88afe31f) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 09:21:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-e51eb67f]) (logid:200c221b) checking health of usage server > 2025-05-28 09:21:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-e51eb67f]) (logid:200c221b) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 09:31:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-ad79b410]) (logid:2ef015d8) checking health of usage server > 2025-05-28 09:31:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-ad79b410]) (logid:2ef015d8) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > 2025-05-28 09:41:47,283 INFO [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-349215d6]) (logid:987a5fd7) checking health of usage server > 2025-05-28 09:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] > (HA-1:[ctx-349215d6]) (logid:987a5fd7) usage server running? false, > heartbeat: Wed May 01 12:00:15 EEST 2024 > ``` > > Lugupidamisega / Best regards, > > Joann Mõndresku > > > > Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee > > WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services > Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee > > ----- Original Message ----- > From: "João Jandre Paraquetti" <j...@scclouds.com.br> > To: "users" <users@cloudstack.apache.org> > Sent: Tuesday, May 27, 2025 5:12:30 PM > Subject: Re: Problem with CloudStack Usage > > Hi Joann, > > I would first check if the Usage server is able to connect to the DB. > Could you confirm that the db.properties file is present at > /etc/cloudstack/usage/ and that it has the correct values? > > Furthermore, could you restart the cloudstack-usage service and check if > any errors/warnings are shown in the logs during the service > initialization? If possible, could you annex the initialization logs for > the Usage server? > > Best regards, > > João Jandre > > On 5/27/25 10:58, Joann Mõndresku wrote: >> Hello. >> >> We tried to enable cloudstack-usage recently, but the management server is >> failing to pair up with it. >> In usage logs, the last line is "UsageServer ready...", no errors witnessed >> either. >> In management server logs, periodically UsageServerMonitorTask claims that >> usage server is not ready. >> Usage and Management are on the same host. >> >> In Management server GUI - Tools -> Usage shows "Server 0" and last >> heartbeat over a year ago with last successful job in 1970 (epoch). >> enable.usage.server is also enabled. >> Any ideas on what we could be doing wrong here? >> >> >> Lugupidamisega / Best regards, >> >> Joann Mõndresku >> >> >> >> Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee >> >> WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services >> Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee >>
