Hi Marcel
I Usually use the zone brand or zadm to start bhyve VM's but I can send
you some working cli's
The illumos-gate builder VM is launched like this here.
619472: /usr/sbin/bhyve -U a03828d6-aaf6-6dec-e075-d3bbc67a737f -H -B
1,manufacturer=Op
argv[0]: /usr/sbin/bhyve
argv[1]: -U
argv[2]: a03828d6-aaf6-6dec-e075-d3bbc67a737f
argv[3]: -H
argv[4]: -B
argv[5]: 1,manufacturer=OpenIndiana,product=OpenIndiana
HVM,version=1.0,serial=a03828d6-aaf6-6dec-e075-d3bbc67a737f,sku=001,family=Virtual
Machine
argv[6]: -c
argv[7]: 8
argv[8]: -m
argv[9]: 16G
argv[10]: -l
argv[11]: bootrom,/usr/share/bhyve/firmware/BHYVE_RELEASE.fd
argv[12]: -s
argv[13]: 0,hostbridge,model=i440fx
argv[14]: -s
argv[15]: 1,lpc
argv[16]: -l
argv[17]: com1,/dev/zconsole
argv[18]: -s
argv[19]: 7:0,ahci-cd,/vm/iso/oibldofgate-cidata.iso
argv[20]: -s
argv[21]: 4:0,virtio-blk,/dev/zvol/rdsk/rpool/vm/oibldofgated0
argv[22]: -s
argv[23]: 6:0,virtio-net-viona,oibldofgate0
argv[24]: oibldofgate
The script that managed that bhyve CLI command is here.
https://github.com/OpenIndiana/pkg5/blob/oi/src/brand/bhyve/init
Installed via system/zones/brand/bhyve and system/bhyve/firmware
Hope this helps
Till
On 15.07.25 08:15, Marcel Telka wrote:
Hi,
I gathered more info, please see below. The observation is that bhyve
allocate kernel memory on the 2nd failing run. It looks like this
memory is lost until the next reboot because the attempt to unload the
vmm module fails (more details below).
BTW, I'm testing this on an up-to-date OpenIndiana.
On Mon, Jul 14, 2025 at 11:22:08PM +0200, Marcel Telka wrote:
Hi,
I'm trying to explore bhyve for the first time and so I tried some basic
example from the bhyve(8) man page. Surprisingly, it does not work.
# reboot
...
# ps -elf | grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 530085 2070 13%
Boot pages 169 0 0%
ZFS File Data 878380 3431 21%
Anon 1339939 5234 32%
Exec and libs 4831 18 0%
Page cache 74943 292 2%
Free (cachelist) 72767 284 2%
Free (freelist) 1261250 4926 30%
Total 4162364 16259
Physical 4162363 16259
# mkfile -n 1g image
# bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
-l com1,stdio -H -P -m 1G vm
virtual machine cannot be booted
# ps -elf | grep bhyve
0 S root 2491 2401 0 73 20 ? 455 ? 08:01:52 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 546514 2134 13%
Boot pages 169 0 0%
ZFS File Data 899780 3514 22%
Anon 1329041 5191 32%
Exec and libs 4802 18 0%
Page cache 78496 306 2%
Free (cachelist) 73948 288 2%
Free (freelist) 1229614 4803 30%
Total 4162364 16259
Physical 4162363 16259
# bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
-l com1,stdio -H -P -m 1G vm
vm exit[0]
reason VMX
rip 0x0000000000000000
inst_length 0
status 0
exit_reason 33 (VM-entry failure due to invalid guest state)
qualification 0x0000000000000000
inst_type 0
inst_error 0
Abort (core dumped)
# ps -elf | grep bhyve
0 R root 2517 2401 0 63 20 ? 455 08:02:14 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 547803 2139 13%
Boot pages 169 0 0%
ZFS File Data 903686 3530 22%
VMM Memory 266240 1040 6%
Anon 1329034 5191 32%
Exec and libs 4802 18 0%
Page cache 78751 307 2%
Free (cachelist) 74013 289 2%
Free (freelist) 957866 3741 23%
Total 4162364 16259
Physical 4162363 16259
# bhyve -c 2 -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,image \
-l com1,stdio -H -P -m 1G vm
bhyve: could not activate CPU 0: Device busy
# ps -elf | grep bhyve
0 S root 2706 2401 0 65 20 ? 455 ? 08:09:56 pts/1
0:00 grep bhyve
# mdb -ke ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 569160 2223 14%
Boot pages 169 0 0%
ZFS File Data 966845 3776 23%
VMM Memory 266240 1040 6%
Anon 1348429 5267 32%
Exec and libs 5712 22 0%
Page cache 79339 309 2%
Free (cachelist) 82921 323 2%
Free (freelist) 843549 3295 20%
Total 4162364 16259
Physical 4162363 16259
# modinfo | grep vmm
189 fffffffff40c0000 35388 307 1 vmm (bhyve vmm)
# modunload -i 189
modunload: can't unload the module: Device busy
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/T0a2ef97770149180-M868c86ca95a068bdbc0985a6
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
