Re: Problem with CloudStack Usage

Wed, 28 May 2025 07:19:32 -0700 

Thanks!

> So I understand correctly this stray entry might be causing the issue
Yes. From what I could gather, Usage is not properly considering stray jobs like this one in its schedule and job take over mechanisms, which results in the Usage job never executing at all.
> and PR #8590 was meant to address it and #10848 would've resolved this automatically?
Both were meant to address issues like this one, but they would not have resolved your specific situation.
I will work on fixing this on a next release. As a workaround for now, could you try deleting this entry from the cloud_usage database and restarting Usage? 

On 5/28/25 10:58, Joann Mõndresku wrote:

Hi.

The query results in this:
```
mysql> select * from cloud_usage.usage_job where end_millis = 0;
+----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
| id | host | pid  | job_type | scheduled | start_millis | end_millis | 
exec_time | start_date | end_date | success | heartbeat           |
+----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
|  1 | NULL |    0 |        1 |         0 |            0 |          0 |         
0 | NULL       | NULL     |    NULL | 2024-05-01 09:00:15 |
+----+------+------+----------+-----------+--------------+------------+-----------+------------+----------+---------+---------------------+
1 row in set (0.00 sec)
```

So I understand correctly this stray entry might be causing the issue and PR 
#8590 was meant to address it and #10848 would've resolved this automatically?
The hostname is cloudstack.wavecom.ee with a load balancer in front on 
different hostname for public consumption.

Lugupidamisega / Best regards,

Joann Mõndresku



Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee

WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services
Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee

----- Original Message -----
From: "Fabricio Duarte" <fabricio.dua...@scclouds.com.br>
To: "users" <users@cloudstack.apache.org>
Sent: Wednesday, May 28, 2025 4:30:37 PM
Subject: Re: Problem with CloudStack Usage

The DB connection is ok, and the error you witnessed is unrelated.

It seems that there are some situations in which an Usage server is
unable to takeover unfinished jobs and execute. This issue may be
related to PRs [1] and [2].

Could you share the result of the following query so that we can
investigate this further?

```
select * from cloud_usage.usage_job where end_millis = 0;
```

Also, what is your hostname?

Thank you.

[1]: https://github.com/apache/cloudstack/pull/10848
[2]: https://github.com/apache/cloudstack/pull/8896

On 5/28/25 03:49, Joann Mõndresku wrote:

Hi.

/etc/cloudstack/usage/db.properties is symlinked to 
/etc/cloudstack/management/db.properties

It has these values defined:
```
db.usage.username=cloud
db.usage.password=ENC(Encrypted Password)
db.usage.host=localhost
db.usage.driver=jdbc:mysql
db.usage.port=3306
db.usage.name=cloud_usage
db.usage.uri=
db.usage.maxActive=100
db.usage.maxIdle=30
db.usage.maxWait=10000
db.usage.url.params=serverTimezone=UTC
db.usage.replicas=localhost,localhost
db.usage.autoReconnect=true
db.usage.failOverReadOnly=false
db.usage.reconnectAtTxEnd=true
db.usage.autoReconnectForPools=true
db.usage.secondsBeforeRetrySource=3600
db.usage.queriesBeforeRetrySource=5000
db.usage.initialTimeout=3600
```
If I change any of the usage db credentials in that file as a test, it will 
cause an error, so it is reading the properties from file seemingly without 
issue.

The only issue in usage.err witnessed:
```
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
```

usage.log pastebin from start to ready: https://pastebin.com/Dw6jy164

Checking with `cat /var/log/cloudstack/management/management-server.log | grep 
"UsageServerMonitorTask"`, every 10min this is shown:
```
2025-05-28 08:41:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) checking health of usage server
2025-05-28 08:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-9e402f81]) (logid:dc8b6ec4) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 08:51:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-d18e3a94]) (logid:4bda154f) checking health of usage server
2025-05-28 08:51:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-d18e3a94]) (logid:4bda154f) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 09:01:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) checking health of usage server
2025-05-28 09:01:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-6fad6aed]) (logid:fa8c9e4c) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 09:11:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-e9020dff]) (logid:88afe31f) checking health of usage server
2025-05-28 09:11:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-e9020dff]) (logid:88afe31f) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 09:21:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-e51eb67f]) (logid:200c221b) checking health of usage server
2025-05-28 09:21:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-e51eb67f]) (logid:200c221b) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 09:31:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-ad79b410]) (logid:2ef015d8) checking health of usage server
2025-05-28 09:31:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-ad79b410]) (logid:2ef015d8) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
2025-05-28 09:41:47,283 INFO  [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-349215d6]) (logid:987a5fd7) checking health of usage server
2025-05-28 09:41:47,284 DEBUG [c.c.h.H.UsageServerMonitorTask] 
(HA-1:[ctx-349215d6]) (logid:987a5fd7) usage server running? false, heartbeat: 
Wed May 01 12:00:15 EEST 2024
```

Lugupidamisega / Best regards,

Joann Mõndresku



Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee

WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services
Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee

----- Original Message -----
From: "João Jandre Paraquetti" <j...@scclouds.com.br>
To: "users" <users@cloudstack.apache.org>
Sent: Tuesday, May 27, 2025 5:12:30 PM
Subject: Re: Problem with CloudStack Usage

Hi Joann,

I would first check if the Usage server is able to connect to the DB.
Could you confirm that the db.properties file is present at
/etc/cloudstack/usage/ and that it has the correct values?

Furthermore, could you restart the cloudstack-usage service and check if
any errors/warnings are shown in the logs during the service
initialization? If possible, could you annex the initialization logs for
the Usage server?

Best regards,

João Jandre

On 5/27/25 10:58, Joann Mõndresku wrote:

Hello.

We tried to enable cloudstack-usage recently, but the management server is 
failing to pair up with it.
In usage logs, the last line is "UsageServer ready...", no errors witnessed 
either.
In management server logs, periodically UsageServerMonitorTask claims that 
usage server is not ready.
Usage and Management are on the same host.

In Management server GUI - Tools -> Usage shows "Server 0" and last heartbeat 
over a year ago with last successful job in 1970 (epoch).
enable.usage.server is also enabled.
Any ideas on what we could be doing wrong here?


Lugupidamisega / Best regards,

Joann Mõndresku



Süsteemiadministraator | Systems Administrator | jo...@wavecom.ee

WaveCom AS | ISO 9001, 27001 & 27017 Certified DC and Cloud services
Endla 16, Tallinn 10142 | www.wavecom.ee | www.facebook.com/wavecom.ee

Reply via email to