On Thu, 28 Mar 2013, Peter Grehan wrote:

Hi Dan,

Any ideas?

Just to confirm: your host system is an i7 3930k, which looks like a 6 core system. Should be plenty beefy enough for a 2 vCPU guest.

That's what I thought, 6 cores, 12 threads, 64 GB RAM.

The issue you are seeing looks a bit like what happens when systems are heavily oversubscribed and vCPUs can't get enough run time.

I was running in on an idle system; only bhyve was running.


How much spare RAM is on the host when you started the vm ? Also, is there much running on the host when the guest is started ?

64GB

Here's some more information (sorry for the formatting):

Case 1:  single vCPU via:
root 64112    1.3  0.0 4219232 25588  1  D+    2:09PM     0:02.47 
/usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 
1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco

Top shows it idle (note low load, 4GB memory given to guest, still 53GB free 
RAM, nothing else clocking CPU cycles):

root@olive:~ # top -bH
last pid: 64148;  load averages:  0.03,  0.22,  0.22  up 0+16:29:31    14:11:51
24 processes:  1 running, 23 sleeping

Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free
ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other
Swap:


  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
64112 root        20    0  4120M 25980K vmidle  2   0:03  0.00% bhyve{coco vcpu 
0}
  925 root        20    0 14296K  2044K select 11   0:00  0.00% syslogd
 2695 root        20    0 81452K  6952K select  2   0:00  0.00% sshd
 2697 root        20    0 20224K  4632K ttyin   1   0:00  0.00% csh
64112 root        52    0  4120M 25980K kqread  9   0:00  0.00% bhyve{coco 
mevent}


So, case #1 all is normal.  I was able to build a kernel with very good 
performance.

The boot sequence looks like the following:

root@olive:/vms/coco # ./vmrun.sh coco
Launching virtual machine "coco" ...
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(r...@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08]
/
  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. Configure Boot [O]ptions...         |  .-                        -.
 |                                         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +-----------------------------------------+


Booting...
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
    r...@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d  Stepping = 
7
  
Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE>
  
Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 5368709120 (5120 MB)
avail memory = 3957534720 (3774 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BHYVE  BVMADT  >
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 1
ioapic0 <Version 1.1> irqs 0-16 on motherboard
kbd0 at kbdmux0
acpi0: <BHYVE BVXSDT> on motherboard
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: no PRT entry for 0.31.INTA
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 
0xc0000000-0xc0001fff at device 1.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
vtnet0: Ethernet address: 00:a0:98:ac:0d:df
virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 
0xc0002000-0xc0003fff at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
vtblk0: 8192MB (16777216 512 byte sectors)
uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0
uart2: console (9600,n,8,1)
Timecounters tick every 10.000 msec
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Timecounter "TSC-low" frequency 1599917752 Hz quality 1000
Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf.
Setting hostid: 0x24bfd025.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% fragmentation)
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Mounting local file systems:.
Writing entropy file:.
Setting hostname: cocopuff.example.com.
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 
1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:ac:0d:df
        inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
        inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 1000baseT <full-duplex>
        status: active
Starting devd.
add net default: gateway 172.22.22.252
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Updating motd:.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.

Fri Mar 29 14:09:20 CDT 2013

FreeBSD/amd64 (cocopuff.example.com) (console)

login:


====== Next, boot the same image with two (2) vCPU:

Command run by vmrun.sh:

 root 64163  182.5  0.1 4219360 48968  1  R+    2:16PM     1:10.42 
/usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 
1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco

Top shows it running out of control:

root@olive:~ # top -bH
last pid: 64191;  load averages:  1.45,  0.52,  0.31  up 0+16:35:12    14:17:32
25 processes:  3 running, 22 sleeping

Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free
ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other
Swap:


  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
64163 root       102    0  4120M   120M CPU2    2   1:01 100.00% bhyve{coco 
vcpu 1}
64163 root       102    0  4120M   120M CPU8    8   1:03 94.48% bhyve{coco vcpu 
0}
15751 root        20    0 81452K  7092K select  5   0:00  0.00% sshd
  807 root        20    0 13112K  4812K select  5   0:00  0.00% devd
  925 root        20    0 14296K  2044K select  9   0:00  0.00% syslogd
 1031 root        20    0 16380K  2400K nanslp 10   0:00  0.00% cron
 2695 root        20    0 81452K  6952K select  2   0:00  0.00% sshd
 2697 root        20    0 20224K  4632K ttyin   1   0:00  0.00% csh
64163 root        52    0  4120M   120M kqread  5   0:00  0.00% bhyve{coco 
mevent}
15753 root        20    0 20224K  4580K pause   1   0:00  0.00% csh

Even though, it's just sitting at the login prompt:

root@olive:/vms/coco # ./vmrun.sh coco
Launching virtual machine "coco" ...
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(r...@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08]
/
  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. Configure Boot [O]ptions...         |  .-                        -.
 |                                         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +-----------------------------------------+


Booting...
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
    r...@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d  Stepping = 
7
  
Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE>
  
Features2=0x83bee217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 5368709120 (5120 MB)
avail memory = 3957534720 (3774 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BHYVE  BVMADT  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 2
ioapic0 <Version 1.1> irqs 0-16 on motherboard
kbd0 at kbdmux0
acpi0: <BHYVE BVXSDT> on motherboard
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: no PRT entry for 0.31.INTA
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 
0xc0000000-0xc0001fff at device 1.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
vtnet0: Ethernet address: 00:a0:98:ac:0d:df
virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem 
0xc0002000-0xc0003fff at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
vtblk0: 8192MB (16777216 512 byte sectors)
uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0
uart2: console (9600,n,8,1)
Timecounters tick every 10.000 msec
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf.
Setting hostid: 0x24bfd025.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% fragmentation)
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
Mounting local file systems:.
Writing entropy file:.
Setting hostname: cocopuff.example.com.
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 
1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:ac:0d:df
        inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
        inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 1000baseT <full-duplex>
        status: active
Starting devd.
add net default: gateway 172.22.22.252
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
syslogd: timed out waiting for child
/etc/rc: WARNING: failed to start syslogd
Clearing /tmp (X related).
Updating motd:.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.

Thu Apr 18 02:06:43 CDT 2013

FreeBSD/amd64 (cocopuff.example.com) (console)

login:


========== end of boot sequence

So, something is broken with SMP on the guest, I'd be happy to help just let me 
know.

Dan

_______________________________________________
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Reply via email to