slavkap opened a new issue, #14585:
URL: https://github.com/apache/cloudstack/issues/14585
### problem
Since VDDK support was added for VMware to KVM migration (#12970, extended
in #13656 and restored in #14256), the KVM agent scans the whole filesystem
many times while it starts up and while it handles ReadyCommand:
`find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1`
On our hosts each scan takes about 20 seconds. The agent runs it repeatedly:
configure() (agent start) -> 2 two scans -> ~47s
initialize() (before StartupRoutingCommand) -> 5 scans -> ~108s
The management server sends ReadyCommand with "wait":"60", but the agent
needs about 107 s to answer it. This sets off a loop that keeps repeating:
1. The agent sends Startup, and the management server sends ReadyCommand.
2. The ReadyAnswer is still not sent when the next ping goes out. The
PingAnswer comes ba("Management server requested startup command to
reinitialize the agent"), so the agentruns initialize() again, which means 5
more scans in parallel.
3. The agent sends a second Startup, and the management server closes the
connection righost connection to host ... Attempting reconnection while we
still have 1 commands inprogress).
4. The agent reconnects and step 1 starts again, roughly every 5 minutes.
The agent never gets to a stable connected state. A full agent restart takes
more than a s its first Startup.
Root cause
In LibvirtComputingResource:
- resolveVddkLibDir() calls detectVddkLibDir() (the find /) whenever
vddkLibDir is blank out VDDK this is always the case, so the (empty) result is
never cached and every callscans again.
- hostSupportsVddk(String) calls resolveVddkLibDir() before it checks
hostSupportsInstancerefore runs even on hosts where virt-v2v isn't installed
(in our log: Failed to detectvirt-v2v password option: Cannot run program
"virt-v2v").
- hostSupportsVddk(), hostSupportsVddkBlockCopy(),
hostSupportsVddkBlockCopyInPlaceFinaliBlockCopyRbd() and
hostSupportsVddkRbdDirectImport() all end up in hostSupportsVddk(). They are
called one after another in initialize() and in LibvirtReadyCommandWrapper, so
each of them runs its own scan.
```
10:18:41,716 DEBUG [cloud.agent.Agent] Sending Startup: Seq 19-0
10:18:42,815 INFO [cloud.agent.Agent] Processing agent ready command, agent
id = 19
10:18:43,403 DEBUG [utils.script.Script] Executing command [/bin/bash -c
find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1 ].
10:19:03,098 DEBUG [utils.script.Script] Successfully executed process [...]
for command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib'
...].
10:19:03,115 DEBUG [utils.script.Script] Executing command [/bin/bash -c
find / -type d -name 'vmware-vix-disklib-distrib' 2>/dev/null | head -n 1 ].
10:19:24,828 DEBUG [utils.script.Script] Successfully executed process [...]
for command [/bin/bash -c find / -type d -name 'vmware-vix-disklib-distrib'
...].
10:19:41,918 DEBUG [cloud.agent.Agent] Received response: Seq 19-1: { ...
"PingAnswer": { ... "sendStartup":"true" ... } }
10:19:41,919 INFO [cloud.agent.Agent] Management server requested startup
command to reinitialize the agent
10:20:29,354 DEBUG [cloud.agent.Agent] ... ReadyAnswer ...
10:21:31,629 DEBUG [cloud.agent.Agent] Sending Startup: Seq 19-3
10:21:31,659 DEBUG [utils.nio.NioClient] Closing socket
Socket[addr=/<mgmt-ip>,port=8250,...]
10:21:36,661 INFO [cloud.agent.Agent] Lost connection to host: <mgmt-ip>.
Attempting reconnection while we still have 1 commands in progress.
10:23:28,740 DEBUG [cloud.agent.Agent] Sending Startup: Seq 0-4
10:24:28,922 INFO [cloud.agent.Agent] Management server requested startup
command to reinitialize the agent
10:26:14,006 DEBUG [utils.nio.NioClient] Closing socket
Socket[addr=/<mgmt-ip>,port=8250,...]
... (the cycle repeats)
```
**Workaround**
None that is reliable. Setting vddk.lib.dir in agent.properties stops the
scans only if it points to a valid VDDK installation (lib64/libvixDiskLib.so*).
An invalid or empty value still falls back to find /.
### versions
- CloudStack: latest main
- Hypervisor: KVM, no VDDK and no virt-v2v installed
### The steps to reproduce the bug
1. Use a KVM host without VMware VDDK (and without virt-v2v), running a
build that includes the latest main
2. Restart cloudstack-agent.
3. Check agent.log: find / runs many times, the ReadyAnswer comes long after
the 60s wait, and the agent keeps reconnecting.
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
