Hi. We tried the workaround and usage server started working. Thank you!
Lugupidamisega / Best regards, Joann Mõndresku Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee ----- Original Message ----- From: "Fabricio Duarte" <fabricio.dua...@scclouds.com.br> To: "users" <users@cloudstack.apache.org> Sent: Wednesday, May 28, 2025 5:18:32 PM Subject: Re: Problem with CloudStack Usage Thanks! > So I understand correctly this stray entry might be causing the issue Yes. From what I could gather, Usage is not properly considering stray jobs like this one in its schedule and job take over mechanisms, which results in the Usage job never executing at all. > and PR #8590 was meant to address it and #10848 would've resolved this automatically? Both were meant to address issues like this one, but they would not have resolved your specific situation. I will work on fixing this on a next release. As a workaround for now, could you try deleting this entry from the cloud_usage database and restarting Usage? On 5/28/25 10:58, Joann Mõndresku wrote: > Hi. > > The query results in this: > ``` > mysql> select * from cloud_usage.usage_job where end_millis = 0; > +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ > | id | host | pid | job_type | scheduled | start_millis | end_millis | > exec_time | start_date | end_date | success | heartbeat | > +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ > | 1 | NULL | 0 | 1 | 0 | 0 | 0 | > 0 | NULL | NULL | NULL | 2024-05-01 09:00:15 | > +----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+ > 1 row in set (0.00 sec) > ``` > > So I understand correctly this stray entry might be causing the issue and PR > #8590 was meant to address it and #10848 would've resolved this automatically? > The hostname is cloudstack.wavecom.ee with a load balancer in front on > different hostname for public consumption. > > Lugupidamisega / Best regards, > > Joann Mõndresku > > > > Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee > > WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services > Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee > > ----- Original Message ----- > From: "Fabricio Duarte" <fabricio.dua...@scclouds.com.br> > To: "users" <users@cloudstack.apache.org> > Sent: Wednesday, May 28, 2025 4:30:37 PM > Subject: Re: Problem with CloudStack Usage > > The DB connection is ok, and the error you witnessed is unrelated. > > It seems that there are some situations in which an Usage server is > unable to takeover unfinished jobs and execute. This issue may be > related to PRs [1] and [2]. > > Could you share the result of the following query so that we can > investigate this further? > > ``` > select * from cloud_usage.usage_job where end_millis = 0; > ``` > > Also, what is your hostname? > > Thank you. > > [1]: https://github.com/apache/cloudstack/pull/10848 > [2]: https://github.com/apache/cloudstack/pull/8896 > > On 5/28/25 03:49, Joann Mõndresku wrote: >> Hi. >> >> /etc/cloudstack/usage/db.properties is symlinked to >> /etc/cloudstack/management/db.properties >> >> It has these values defined: >> ``` >> db.usage.username=cloud >> db.usage.password=ENC(Encrypted Password) >> db.usage.host=localhost >> db.usage.driver=jdbc:mysql >> db.usage.port=3306 >> db.usage.name=cloud_usage >> db.usage.uri= >> db.usage.maxActive=100 >> db.usage.maxIdle=30 >> db.usage.maxWait=10000 >> db.usage.url.params=serverTimezone=UTC >> db.usage.replicas=localhost,localhost >> db.usage.autoReconnect=true >> db.usage.failOverReadOnly=false >> db.usage.reconnectAtTxEnd=true >> db.usage.autoReconnectForPools=true >> db.usage.secondsBeforeRetrySource=3600 >> db.usage.queriesBeforeRetrySource=5000 >> db.usage.initialTimeout=3600 >> ``` >> If I change any of the usage db credentials in that file as a test, it will >> cause an error, so it is reading the properties from file seemingly without >> issue. >> >> The only issue in usage.err witnessed: >> ``` >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >> SLF4J: Defaulting to no-operation (NOP) logger implementation >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further >> details. >> ``` >> >> usage.log pastebin from start to ready: https://pastebin.com/Dw6jy164 >> >> Checking with `cat /var/log/cloudstack/management/management-server.log | >> grep "UsageServerMonitorTask"`, every 10min this is shown: >> ``` >> 2025-05-28 08:41:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) checking health of usage server >> 2025-05-28 08:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 08:51:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-d18e3a94]) (logid:4bda154f) checking health of usage server >> 2025-05-28 08:51:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-d18e3a94]) (logid:4bda154f) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 09:01:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) checking health of usage server >> 2025-05-28 09:01:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 09:11:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-e9020dff]) (logid:88afe31f) checking health of usage server >> 2025-05-28 09:11:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-e9020dff]) (logid:88afe31f) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 09:21:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-e51eb67f]) (logid:200c221b) checking health of usage server >> 2025-05-28 09:21:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-e51eb67f]) (logid:200c221b) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 09:31:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-ad79b410]) (logid:2ef015d8) checking health of usage server >> 2025-05-28 09:31:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-ad79b410]) (logid:2ef015d8) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> 2025-05-28 09:41:47,283 INFO [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-349215d6]) (logid:987a5fd7) checking health of usage server >> 2025-05-28 09:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] >> (HA-1:[ctx-349215d6]) (logid:987a5fd7) usage server running? false, >> heartbeat: Wed May 01 12:00:15 EEST 2024 >> ``` >> >> Lugupidamisega / Best regards, >> >> Joann Mõndresku >> >> >> >> Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee >> >> WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services >> Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee >> >> ----- Original Message ----- >> From: "João Jandre Paraquetti" <j...@scclouds.com.br> >> To: "users" <users@cloudstack.apache.org> >> Sent: Tuesday, May 27, 2025 5:12:30 PM >> Subject: Re: Problem with CloudStack Usage >> >> Hi Joann, >> >> I would first check if the Usage server is able to connect to the DB. >> Could you confirm that the db.properties file is present at >> /etc/cloudstack/usage/ and that it has the correct values? >> >> Furthermore, could you restart the cloudstack-usage service and check if >> any errors/warnings are shown in the logs during the service >> initialization? If possible, could you annex the initialization logs for >> the Usage server? >> >> Best regards, >> >> João Jandre >> >> On 5/27/25 10:58, Joann Mõndresku wrote: >>> Hello. >>> >>> We tried to enable cloudstack-usage recently, but the management server is >>> failing to pair up with it. >>> In usage logs, the last line is "UsageServer ready...", no errors witnessed >>> either. >>> In management server logs, periodically UsageServerMonitorTask claims that >>> usage server is not ready. >>> Usage and Management are on the same host. >>> >>> In Management server GUI - Tools -> Usage shows "Server 0" and last >>> heartbeat over a year ago with last successful job in 1970 (epoch). >>> enable.usage.server is also enabled. >>> Any ideas on what we could be doing wrong here? >>> >>> >>> Lugupidamisega / Best regards, >>> >>> Joann Mõndresku >>> >>> >>> >>> Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee >>> >>> WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services >>> Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee >>>
