GitHub user TadiosAbebe added a comment to the discussion: Degraded cloudstack agent
@bradh352 , opening the console is also my first step when checking whether a compute host is experiencing issues. Over the past few weeks, I ran several tests across my five compute hosts (all using Ceph) to better understand the behavior. ### Initial Tests These tests were performed a few days after restarting both cloudstack-agent and libvirtd. | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 86 | 30 | 60 | 5| | host2 | 37 | 18 | 28 | 4 | | host3 | 54 | 9 | 33 | 1 | | host4 | 49 | 3 | 19 | 0 | | host5 | 12 | 3 | 19 | 0 | ### After Restarting libvirtd I restarted libvirtd on all hosts and repeated the same tests. | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 21 | 3 | 19 | 0| | host2 | 22 | 3 | 19 | 0 | | host3 | 25 | 3 | 19 | 0 | | host4 | 21 | 7 | 19 | 0 | | host5 | 22 | 3 | 18 | 0 | ### Same Tests Again (After ~2 Days) | host | vm_create(in sec) | vm_stop(in sec) | vm_start(in sec) | vm_console(in sec) | |---|---|---|---|---| | host1 | 31 | 9 | 21 | 1| | host2 | 28 | 3 | 18 | 0 | | host3 | 28 | 6 | 23 | 1 | | host4 | 30 | 3 | 19 | 0 | | host5 | 26 | 3 | 18 | 0 | These results may not be perfectly accurate due to different factors, but the performance differences are still notable especially the improvement immediately after restarting libvirtd. My main concern with scheduling an automated restart of cloudstack-agent was, if it restarts in the middle of an operation, it may leave behind orphaned resources. I believe this is how I ended up with volumes stuck in Destroy state and a virtual router stuck in Expunging, which I’m still unable to fully remove. It would be helpful to know whether restarting libvirtd is truly the reliable fix for you too. Even if that’s the case, I still don’t understand why this degradation happens over time. GitHub link: https://github.com/apache/cloudstack/discussions/12450#discussioncomment-15515825 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
