Hello.
> The cloudstack-guest-tool could be expanded to add this functionality to
> make it easier, but I doubt if we want API access as its also very
> dangerous to be able to execute commands inside all the VMs.
+1 on the doubt. We run a small public cloud on CloudStack/KVM with
customer-owned VMs and hit exactly this question a month ago. We ended up
NOT exposing guest-exec at all, and splitting the two use cases it normally
gets abused for. Sharing the pattern and the gotchas we paid for, since a
few of them are not documented anywhere I could find.
1) What you can do today, without any CloudStack API
----------------------------------------------------
On the host running the instance (listVirtualMachines gives you the
'hostname' field, and the domain is the i-<account>-<id>-VM name):
virsh qemu-agent-command i-2-345-VM \
'{"execute":"guest-exec","arguments":{"path":"/bin/uname",
"arg":["-a"],"capture-output":true}}'
-> {"return":{"pid":1234}}
virsh qemu-agent-command i-2-345-VM \
'{"execute":"guest-exec-status","arguments":{"pid":1234}}'
That is the whole mechanism. It is asynchronous (exec returns a pid, status
returns base64 output) and it needs root on the hypervisor. There is no API
call in front of it, which - as you say - is arguably a feature.
2) What we bake into every template instead
-------------------------------------------
We build all our CloudStack templates offline with virt-customize from
upstream cloud images, and every template carries the same guest layer.
The relevant bit: qemu-guest-agent is installed, but guest-exec* and
guest-file-* are BLOCKED. We keep guest-ssh-* and guest-info. That draws
the line at "metadata plus a break-glass SSH key" rather than "arbitrary
root inside a tenant VM".
Support access then looks like this, and only this:
virsh qemu-agent-command <domain> \
'{"execute":"guest-ssh-add-authorized-keys","arguments":
{"username":"debian","keys":["ssh-ed25519 AAAA... support"]}}'
...ssh in, do the work, then guest-ssh-remove-authorized-keys. It is
visible to the customer, reversible, and every use is recorded in the
ticket. A very different security story from a silent root exec.
Two gotchas that cost us real time when making the blocklist actually
stick (verified in the distro package sources, not assumed):
* Debian / Ubuntu / SUSE: the mechanism that works is the qemu-ga config
file, /etc/qemu/qemu-ga.conf:
[general]
block-rpcs=guest-exec,guest-exec-status,guest-file-open,guest-file-read,guest-file-write,guest-file-close,guest-file-seek,guest-file-flush
The systemd unit does NOT read /etc/default/qemu-guest-agent. That file
is a sysvinit relic, it is still shipped, and editing it looks like it
worked. We delete it in our templates so nobody is misled by it.
* RHEL / Rocky: the unit reads /etc/sysconfig/qemu-ga, variable
FILTER_RPC_ARGS (not BLOCK_RPCS). The shipped default there is an
--allow-rpcs= list which ALSO excludes guest-ssh-*, so if you leave it
alone your key-injection path is dead too. We replace the whole line
with FILTER_RPC_ARGS="--block-rpcs=<same list as above>".
(--allow-rpcs and --block-rpcs are mutually exclusive.)
We assert the block in an offline CI gate on the built image, before the
template is ever registered.
3) A second channel instead of guest-exec
-----------------------------------------
For the "I need to see what is going on inside that VM" case - which is
what guest-exec usually degenerates into - we add a second virtio-serial
channel and a ~100-line read-only agent listening on it. Closed verb
dictionary: snapshot | psi | topio | topnet, JSON out. Loadavg, a 15 min
ring buffer of /proc/pressure (PSI), top processes by IO, established
connections and conntrack count. No exec, no file access, and the systemd
unit is capped (CPUQuota=5%, MemoryMax=64M, Nice=19, IO idle) so the
diagnostic agent can never become the noisy neighbour it is meant to find.
The point is that the reviewable surface is a fixed set of read-only verbs
instead of a shell. It is deliberately far less powerful than GuestExec.
4) Getting CloudStack to attach that channel - the fiddly part
--------------------------------------------------------------
CloudStack will not add a non-standard channel on its own. Two routes:
(a) extraconfig. Requires, at Account scope:
enable.additional.vm.configuration = true
allow.additional.vm.configuration.list.kvm =
devices,channel,source,target,address
Every nested tag has to be listed. Ours was empty by default, and with
an empty list CloudStack silently drops the configuration - the VM
boots fine, you just have no channel. Warning in the other direction:
turning the setting on with an INCOMPLETE list can prevent VMs from
starting, so test on one instance first.
The snippet is appended as a child of <domain>, so it must be wrapped
in its own <devices> block:
<devices><channel type="unix"><target type="virtio"
name="io.example.support.0"/></channel></devices>
We put that in the TEMPLATE details at registration time
(details[0].extraconfig-1=...), so every VM created from the template
gets the channel without anyone passing anything at deploy time.
The one that cost us a day: the extraconfig API PARAMETER is
URL-decoded on save, but a template DETAIL value is copied verbatim.
URL-encode the XML into the template detail and the guest ends up with
the literal string %3Cdevices%3E..., libvirt ignores it, the VM boots
perfectly normally, and the failure is completely silent. Measured on
three VMs: extraconfig as API param -> channel present; url-encoded
template detail -> absent; raw template detail -> present. Store it
raw; the HTTP transport encodes it once for you.
(b) Per-host Groovy hook libvirt-vm-xml-transformer (agent.hooks.basedir in
agent.properties), if you would rather do it wholesale per host than
per template.
Either route coexists with org.qemu.guest_agent.0, which CloudStack adds
unconditionally since 4.10 - you end up with both ports under
/dev/virtio-ports/ in the guest.
5) Honest limitations
---------------------
* Reading the channel still needs root on the hypervisor (socat against
the libvirt channel socket under /var/lib/libvirt/qemu/channel/target/).
There is no API for it and, so far, no automated consumer on our side
either - today it is an operator triage tool, not a product feature.
* It does not help you push configuration into a VM. For that we stayed
with cloud-init and the VR (set_passwords: always, so the CloudStack
password reset keeps working on every boot).
So, concretely on the design question in this thread: I agree that an API
for guest-exec would be too much. But an API in front of a FIXED, read-only
verb set - roughly what cloudstack-guest-tool could grow into - would be
genuinely useful and reviewable, and it covers a large share of what
operators actually reach for GuestExec to do.
Regards,
Piotr
-----Original Message-----
From: Wido den Hollander via users <[email protected]>
Sent: Tuesday, September 8, 2026 8:11 PM
To: [email protected]; Sanjay Kumar <[email protected]>;
[email protected]
Cc: Wido den Hollander <[email protected]>
Subject: Re: is there any GuestExec functionality like in apache cloudstack?
Op 08-09-2026 om 18:34 schreef Sanjay Kumar:
> Hello!
>
> On VMWARE we use the GuestExec functionality. I looked through the API
> and don't see any similar way to run commands on the host through
> CloudStack. Is there a way to add custom plugins that could run on the
> hypervisor? Since my infra is KVM Based, I would presume if we
> switched from open-vm-tools to qemu-guest-agent then we should VIRSH
> to run commands. However, a quick glance didn't reveal any supported
> way to run that VIRSH on the hypervisor that is running the machine itself.
>
Good question! I just opened a PR [0] to enhance the 'cloudstack-guest-tool' on
the KVM hypervisors, it now allows for changing passwords.
The Qemu Guest Agent has support to execute commands (as root) inside the VM,
but there is no API-call in CloudStack to do this. You would need to have root
access to the hypervisor.
The cloudstack-guest-tool could be expanded to add this functionality to make
it easier, but I doubt if we want API access as its also very dangerous to be
able to execute commands inside all the VMs.
Wido
[0]: https://github.com/apache/cloudstack/pull/14080
>
> Any help would be really appreciated.
>
> Thank you
