GitHub user wverleger created a discussion: virtual router - resources / system
load
Hi,
running ACS 4.22.0 with KVM hypervisors (ubuntu24 and alma linux 9.6).
Before upgrading to ACS 4.22.0 from 4.19.3.0 we had a VPC with redundant
routers (4 vCPU / 1G memory each).
This VPC produces a lot of traffic and to secure the sessions between the
routers we scaled them up to 4 vCPU / 1G memory.
After upgrading to ACS 4.22.0 we encountered perfomance problems within the
VPC. After disabling redundant routers and scaling up the remaining router to:
8 vCPU / 4G memory the system stabilised.
On the hypervisor (ubuntu 24.04) we see a high system load (I guess I/O
interrupts, scheduler, etc) for this qemu process:
```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7479 root 20 0 9920.3m 857580 23040 R 96.7 0.1 25,56 CPU 3/KVM
7483 root 20 0 9920.3m 857580 23040 R 94.3 0.1 25,06 CPU 7/KVM
7481 root 20 0 9920.3m 857580 23040 R 89.0 0.1 23,07 CPU 5/KVM
7477 root 20 0 9920.3m 857580 23040 R 88.7 0.1 23,13 CPU 1/KVM
7476 root 20 0 9920.3m 857580 23040 R 81.3 0.1 20,43 CPU 0/KVM
7480 root 20 0 9920.3m 857580 23040 R 78.7 0.1 20,54 CPU 4/KVM
7478 root 20 0 9920.3m 857580 23040 R 65.0 0.1 16,52 CPU 2/KVM
7482 root 20 0 9920.3m 857580 23040 S 58.7 0.1 15,12 CPU 6/KVM
7474 root 20 0 9920.3m 857580 23040 R 58.0 0.1 15,18
vhost-7464
7473 root 20 0 9920.3m 857580 23040 R 44.0 0.1 10,21
vhost-7464
7470 root 20 0 9920.3m 857580 23040 R 32.0 0.1 8,46
vhost-7464
7472 root 20 0 9920.3m 857580 23040 S 7.7 0.1 134:28.09
vhost-7464
7471 root 20 0 9920.3m 857580 23040 S 5.3 0.1 85:29.00
vhost-7464
7464 root 20 0 9920.3m 857580 23040 S 0.3 0.1 0:30.19
qemu-system-x86
7468 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.01
qemu-system-x86
7469 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:02.24
vhost-7464
7475 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:04.83 IO
mon_iothread
7486 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00
vnc_worker
7488 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00
kvm-nx-lpage-re
340663 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340701 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340737 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340738 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340739 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340740 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340741 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
340742 root 20 0 9920.3m 857580 23040 S 0.0 0.1 0:00.00 worker
```
While the router vm itself is mostly idling within the vm:
```
Tasks: 150 total, 1 running, 149 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 99.2 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 98.8 id, 0.0 wa, 0.0 hi, 1.2 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 1.9 si, 0.0 st
%Cpu4 : 0.8 us, 0.4 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 1.5 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 2.1 si, 0.0 st
MiB Mem : 3911.8 total, 3436.8 free, 448.7 used, 122.2 buff/cache
MiB Swap: 487.0 total, 487.0 free, 0.0 used. 3463.0 avail Mem
```
On other hypervisors (alma linux 9.6) we see routers idling both on hypervisor
and within the router.
1. Is there any known change regarding virtual routers from 4.19.3.0 to 4.22.0
we are overlooking?
2. Could this be an expected behaviour (high hypervisor load, low system load
in router)?
Thank you!
GitHub link: https://github.com/apache/cloudstack/discussions/12739
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
