Hi Chi,
You may have noticed, there are some discussions on this topic, including a
bug fix and some global settings.
I think you can wait for the upcoming 4.20.1.0 release which contains
several bug fixes.
I tested 4.20.1.0-SNAPSHOT by deploying 80 vm instances, it looks good.
below are the list of vms
mysql> select id,name,state,created,removed,update_time from vm_instance
where id >=1259;
+------+-----------------------------------------+---------+---------------------+---------+---------------------+
| id | name | state | created
| removed | update_time |
+------+-----------------------------------------+---------+---------------------+---------+---------------------+
| 1259 | VM-9c22c201-da71-4f1f-ad4e-7e54e6e0d070 | Running | 2025-03-21
09:39:44 | NULL | 2025-03-21 09:41:51 |
| 1260 | VM-21bef6e6-8a07-4a73-8422-ea56078a36eb | Running | 2025-03-21
09:39:44 | NULL | 2025-03-21 09:40:22 |
| 1261 | VM-adf69a50-5072-47f0-be94-5c24082ae95f | Running | 2025-03-21
09:39:47 | NULL | 2025-03-21 09:41:05 |
| 1262 | VM-ce501a62-855c-43b4-be77-414f739645b9 | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:40:27 |
| 1263 | VM-cc999859-e48a-4ff0-92c1-c057c7b3a665 | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:43:01 |
| 1264 | VM-42c3442a-ae75-4bac-ad05-f101c0b961c8 | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:42:34 |
| 1265 | VM-13e5b030-fd6b-4bbd-8356-5e71ad54ee2b | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:42:07 |
| 1266 | VM-c8691163-1034-4e12-9937-d6da244171eb | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:43:11 |
| 1267 | VM-033b1e9f-8c73-4cfe-8cb3-67495b194118 | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:44:05 |
| 1268 | VM-a9ed3742-fbc6-4bea-9af6-9bb6eec3c5a1 | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:46:24 |
| 1269 | VM-f5faa276-4275-431a-b5e9-925119a2f29c | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:46:00 |
| 1270 | VM-f7dfa067-f288-4f50-a2a3-31c01d93f73e | Running | 2025-03-21
09:39:51 | NULL | 2025-03-21 09:43:04 |
| 1271 | VM-e5eeb5d1-564c-4e35-8d7c-5d707213542b | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:41:38 |
| 1272 | VM-9c3f1527-1341-4b19-9a1f-de0de3774ed1 | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:46:16 |
| 1273 | VM-90dce95a-e650-482f-a19b-cdebd8d126fe | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:42:02 |
| 1274 | VM-249534d5-7574-4a94-aeb5-543dd0f7507a | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:46:10 |
| 1275 | VM-25a7b77c-c401-41d9-83f2-da5a1be89425 | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:44:39 |
| 1276 | VM-a968c677-4354-4db2-ac6e-4a661ceeacb6 | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:45:58 |
| 1277 | VM-e5163ea3-bacf-4c36-8251-45fee35b0b08 | Running | 2025-03-21
09:39:52 | NULL | 2025-03-21 09:44:43 |
| 1278 | VM-b1a4f873-fcbb-43f8-8a3a-b3ffc6303c6f | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:46:13 |
| 1279 | VM-0d3c4d9b-8473-4ade-9eab-6a5296fa3a01 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:44:57 |
| 1280 | VM-56f5b7ea-26c1-41b5-9ad4-86334e3810e6 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:46:01 |
| 1281 | VM-eec1e66d-fafd-4d84-bb1b-d6f329e29338 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:44:45 |
| 1282 | VM-e3631409-ab65-444a-8e00-591aae1d6ca4 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:45:57 |
| 1283 | VM-d358dbaf-118b-4e24-bf23-bb068df9a1eb | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:42:21 |
| 1284 | VM-443fd974-3c43-40a6-beba-dd160cd50909 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:44:42 |
| 1285 | VM-7532a299-e387-429a-8bf3-8bde37e34453 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:42:00 |
| 1286 | VM-730ddc99-acb6-4518-a63f-6a824ad653dc | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:44:52 |
| 1287 | VM-125250b6-2774-40a5-967f-d6ebd4d7a70d | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:42:25 |
| 1288 | VM-e8758675-afd8-4a05-9d8b-3116c62377d4 | Running | 2025-03-21
09:39:53 | NULL | 2025-03-21 09:46:20 |
| 1289 | VM-17520053-a318-4857-9ad3-c67de8fc235f | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:44:54 |
| 1290 | VM-64c2f22e-5ba2-4bd8-8826-24477acc7bfe | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:44:59 |
| 1291 | VM-875a080e-c628-4540-b3be-000242422be7 | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:45:00 |
| 1292 | VM-df94fbab-778b-4767-a676-d4681a8f3407 | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:45:20 |
| 1293 | VM-9f0498fe-e8c1-4322-b778-d365c0712b6b | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:44:48 |
| 1294 | VM-df14bdb9-448f-460a-89df-76cccf402a20 | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:45:00 |
| 1295 | VM-48186175-5328-4176-ac29-abe9352ff454 | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:45:10 |
| 1296 | VM-54e0b51a-233c-4a27-be5b-b0a46cbbe59c | Running | 2025-03-21
09:39:54 | NULL | 2025-03-21 09:45:05 |
| 1297 | VM-1d470263-6345-4b1e-8a51-fa0821fb2e4f | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:46:33 |
| 1298 | VM-64c3c580-4891-4ab4-bfa5-14b5520bf8b7 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:44:51 |
| 1299 | VM-14cc2f0e-942b-446f-8519-2f42abc641f4 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:45:07 |
| 1300 | VM-6d62bcd5-fe4a-425b-8bca-7091a74a4e80 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:42:33 |
| 1301 | VM-7762687f-acc4-4eda-9357-18042285c198 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:45:14 |
| 1302 | VM-519a6d4c-614f-420f-af0b-12ddc2f1ba0a | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:46:41 |
| 1303 | VM-932249f1-2908-4bf1-982f-5744cfbaefae | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:45:13 |
| 1304 | VM-676b677d-f0fd-4a59-8a79-47f7c49c0c49 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:45:07 |
| 1305 | VM-dcec2429-306e-41e2-99df-6206d5bf087b | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:43:04 |
| 1306 | VM-84765edf-f877-4671-9190-36b0b26ef892 | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:45:23 |
| 1307 | VM-b8df91a7-36a4-479b-be45-e0738eac257c | Running | 2025-03-21
09:39:55 | NULL | 2025-03-21 09:42:46 |
| 1308 | VM-e27d2a59-ff62-4b77-aff8-0f59dfbe5536 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:45:25 |
| 1309 | VM-3e37ab42-ba2e-4886-b491-5d24b30a9da6 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:46:57 |
| 1310 | VM-26542017-55b1-4778-ba66-89cce970a9d0 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:46:35 |
| 1311 | VM-3aa450b0-18e8-42f4-b497-81ae100f0351 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:46:36 |
| 1312 | VM-8d3e9f47-ed55-4881-8754-a35f58dd91b6 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:42:31 |
| 1313 | VM-30865bb9-4ed5-4c93-9ba5-b57f0f32afd8 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:45:27 |
| 1314 | VM-e47b769f-0323-4f5f-9407-06b6f72ee429 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:45:32 |
| 1315 | VM-7e47e759-b1ff-4395-af12-19d90a5142da | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:44:24 |
| 1316 | VM-a6e87953-9212-461e-ad2a-ca4e464b641c | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:45:18 |
| 1317 | VM-8945ec20-eddf-4d04-a300-3e5d2e4fd352 | Running | 2025-03-21
09:39:56 | NULL | 2025-03-21 09:45:34 |
| 1318 | VM-f7f93149-a509-4c09-bb30-aed1c73465af | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:42:54 |
| 1319 | VM-9f9ecdbd-5fa8-41ac-8bb5-dcf44290b730 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:45:32 |
| 1320 | VM-9dda1778-91cb-464d-9068-bc2402976a23 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:43:02 |
| 1321 | VM-d75f7216-1e9a-4172-8421-ea810207e981 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:45:43 |
| 1322 | VM-9165c1cd-ce6b-42e6-b037-e365ad5980e1 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:43:26 |
| 1323 | VM-74704ab0-242f-407f-91d9-12370b411534 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:45:59 |
| 1324 | VM-707ffcce-9182-45e4-b4e0-49debff01214 | Running | 2025-03-21
09:39:57 | NULL | 2025-03-21 09:45:44 |
| 1325 | VM-a5548fc1-3bef-4afa-80fd-0854a558f608 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:46:07 |
| 1326 | VM-4b49dabf-2f22-437e-bf7e-6ceca7f09b9e | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:43:12 |
| 1327 | VM-52200de7-0498-4fc4-9552-558524b9f8a5 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:45:39 |
| 1328 | VM-f214386a-7092-4b55-849a-41366d8b45a5 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:43:21 |
| 1329 | VM-99b69515-9740-42d6-ba3f-818796ac07d9 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:46:26 |
| 1330 | VM-633e5740-9fe5-4808-a759-64852626ef07 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:45:54 |
| 1331 | VM-916a0977-7631-4949-b5ec-d1cfb0cd6a88 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:45:36 |
| 1332 | VM-7bb0be9b-1e7c-4891-90c1-27f3bbd67f31 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:43:19 |
| 1333 | VM-257ba10d-584a-4aba-82d2-e680fb0b5843 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:43:50 |
| 1334 | VM-291ea32c-73aa-482a-9ebe-de87ff15bfe9 | Running | 2025-03-21
09:39:58 | NULL | 2025-03-21 09:44:44 |
| 1335 | VM-9c37b0c2-efcc-4613-9f67-564ffe321901 | Running | 2025-03-21
09:39:59 | NULL | 2025-03-21 09:47:01 |
| 1336 | VM-26750ca6-9359-4171-8498-d44699e5aec3 | Running | 2025-03-21
09:39:59 | NULL | 2025-03-21 09:46:40 |
| 1337 | VM-097bb91b-46dc-41ae-b2a4-d557001b1533 | Running | 2025-03-21
09:39:59 | NULL | 2025-03-21 09:43:38 |
| 1338 | VM-c0f8add3-975a-4534-8508-15af17030fc5 | Running | 2025-03-21
09:39:59 | NULL | 2025-03-21 09:43:46 |
+------+-----------------------------------------+---------+---------------------+---------+---------------------+
80 rows in set (0.00 sec)
btw: I do not see a difference with the upgrade from 4.18->4.19->4.20 and
pure 4.20 installation.
-Wei
On Thu, Mar 20, 2025 at 1:45 PM Chi vediamo <tatay...@gmail.com> wrote:
> I support what Alex is requesting.
>
>
> - If you are happy with some older CloudStack version, please also share
> - If default configuration is not suitable for dozens of VMs, please
> advise on config changes. If there is any document on this, I'd be glad to
> work through it.
>
>
> Yes, please indicate What changes needs to be made fto make Cloudstack
> 4.20 production ready as a Standalone not as an Upgrade, seems there are
> too many changes -missing parameteres, links, or changed parameters - to
> the default configurations, that VMs are just not coming up after 2 or 3
> VMs. Even using massive servers.
>
> Please share notes and or documentation about. If I have to bring 200 VMs
> at once. How are you validating this actually works?.
>
> I personally readed, and follow tons of dicussions. I can share the only
> think I found Is you have to install rel 4.18, then rel 4.19 and the 4.20,
> and you will have less issues with the 4.20 - is way better, but Still
> can't be installed as a standalone.
>
> 4.20 as a standalone just does not work for us. I have to ask again,
> should 4.20 work as Stand Alone?
>
>
> Tata Y.
>
> > On Mar 20, 2025, at 6:44 AM, akrasnov-drv (via GitHub) <g...@apache.org>
> wrote:
> >
> >
> > GitHub user akrasnov-drv closed a discussion: CloudStack starts
> misbehaving under minimal load
> >
> > _I opened several other discussions/issues for my scale problems with
> CloudStack (partially solved). But now I managed to simplify the env, and
> want to start it clean as a new discussion_
> > <br>
> > ### My env
> > ```
> > CloudStack 4.20.0.0
> > on Ubuntu 22.04.5 LTS (amd64)
> > with libvirt 8.0.0-1ubuntu7.10
> > OpenJDK 64-Bit Server VM (build 17.0.13+11-Ubuntu-2ubuntu122.04, mixed
> mode, sharing)
> > ```
> > <br>
> > I configured basic shared network, IPs are from my /24 internal subnet.
> VRouter serves as dhcp, metadata and dns server, but VMs are directly
> accessible from network and have direct outside access (via subnet gateway).
> > I left single host to run VMs, separate from CloudStack management host.
> > VM host can run about 20 VMs.
> > Network is 1 Gbps.
> > Primary and secondary storage are in NFS from another host in the same
> subnet.
> > <br>
> >
> > ### The task
> > I'm trying to start 20 VMs from the same qcow image and the same compute
> offering using local disk (on RAID-0 of 2 SSDs)
> > There is several seconds delay between VM deployments.
> > <br>
> >
> > ### The problem
> > 2-3 first VMs start fine. Then VMs in `starting` state begin
> accumulating.
> > 2 running + 1 starting
> > 3 running + 2 starting
> > Finally I get about 7 running and 13 starting and that state persists. I
> mean even after some 15 minutes those starting are still starting.
> > To make sure, I see just those running ones in virsh output. All
> "starting" ones are not even created.
> > When I tried to start more (in hope to see them in virsh), they just
> joined to the pool of "starting" ones (at the moment I see about 28 in
> "starting" state on the host that can run just 20)
> >
> > But that's just a part of problem.
> > The second part is that from those 7 running just 3-5 started well, and
> the rest did not get hostname from VR metadata server.
> > Sometimes I also see VMs that do not get IP, but that's more rare. For
> such ones reboot/reset does not help, but reinstall via UI button usually
> fixes the problem.
> >
> > Logs on the agent side are quite idle. Just periodically I see messages
> like this
> > ```
> > 2025-02-17 18:03:09,959 WARN [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-2:[]) (logid:402faf9c) Groovy script
> '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not
> available. Transformations will not be applied.
> > 2025-02-17 18:03:09,959 WARN [kvm.resource.LibvirtKvmAgentHook]
> (agentRequest-Handler-2:[]) (logid:402faf9c) Groovy scripting engine is not
> initialized. Data transformation skipped.
> > ```
> > Management log is active.
> > In these tests I did not feel UI slowdown as it was when I had more
> hosts connected with advanced network. Nevertheless I was not able to start
> properly VMs I wanted.
> > <br>
> >
> > ### My questions
> > - Lately I've got an impression that the last CloudStack versions are
> imperfect. Would be great if someone could share positive experience of
> running dozens of VMs with later versions.
> > - If you are happy with some older CloudStack version, please also share
> > - If default configuration is not suitable for dozens of VMs, please
> advise on config changes. If there is any document on this, I'd be glad to
> work through it.
> >
> > I continue to experiment and am able to share logs if required.
> > All useful information is appreciated.
> >
> > Thanks,
> > Alex.
> >
> >
> >
> >
> > GitHub link: https://github.com/apache/cloudstack/discussions/10414
> >
> > ----
> > This is an automatically sent email for users@cloudstack.apache.org.
> > To unsubscribe, please send an email to:
> users-unsubscr...@cloudstack.apache.org
> >
>
>
