Hi Dan,

On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack <m...@macktronics.com> wrote:

>
>
> Here you go Neel:
>
> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0
> vcpu0
> vm exits due to external interrupt      45051
> number of times hlt was intercepted     2277
> number of NMIs delivered to vcpu        0
> vcpu total runtime                      105001652482
> number of ticks vcpu was idle           919
> vcpu migration across host cpus         13
> number of times hlt was ignored         0
> total number of vm exits                9240925
> root@olive:~ #
> root@olive:~ #
> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1
> vcpu1
> vm exits due to external interrupt      149431
> number of times hlt was intercepted     4222
> number of NMIs delivered to vcpu        0
> vcpu total runtime                      106876622528
> number of ticks vcpu was idle           869
> vcpu migration across host cpus         20
> number of times hlt was ignored         0
> total number of vm exits                8065909
>
>
That did not help a whole lot because we were not keeping track of all the
reasons a vcpu could exit.

I have fixed that in r248935:
http://svnweb.freebsd.org/base?view=revision&revision=248935

Do you mind updating your vmm.ko with the change and getting the stats
again?

best
Neel


> Dan
>
>
>
> On Fri, 29 Mar 2013, Neel Natu wrote:
>
>  Hi Dan,
>>
>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack <m...@macktronics.com> wrote:
>>
>>
>>> I ran the procsystime dtrace script on bhyve when it was chewing up all
>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime:
>>>
>>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT
>>>
>>> Tracing... Hit Ctrl-C to end...
>>> dtrace: 158536 dynamic variable drops with non-empty dirty list
>>> dtrace: 207447 dynamic variable drops with non-empty dirty list
>>> dtrace: 189205 dynamic variable drops with non-empty dirty list
>>> dtrace: 164341 dynamic variable drops with non-empty dirty list
>>> dtrace: 246307 dynamic variable drops with non-empty dirty list
>>> dtrace: 187640 dynamic variable drops with non-empty dirty list
>>> dtrace: 214771 dynamic variable drops with non-empty dirty list
>>> dtrace: 221265 dynamic variable drops with non-empty dirty list
>>> ^C
>>> dtrace: 243468 dynamic variable drops with non-empty dirty list
>>>
>>> Elapsed Times for processes bhyve,
>>>
>>>          SYSCALL          TIME (ns)
>>>         _umtx_op              18349
>>>           writev             135712
>>>           preadv           16175267
>>>          pwritev           22924378
>>>            ioctl         4353897920
>>>           TOTAL:         4393151626
>>>
>>> CPU Times for processes bhyve,
>>>
>>>          SYSCALL          TIME (ns)
>>>         _umtx_op               8815
>>>           writev             103145
>>>          pwritev           10647023
>>>           preadv           15159976
>>>            ioctl         3943399889
>>>           TOTAL:         3969318848
>>>
>>> Syscall Counts for processes bhyve,
>>>
>>>          SYSCALL              COUNT
>>>         _umtx_op                 17
>>>           writev                 43
>>>          pwritev                639
>>>           preadv               1183
>>>            ioctl             652782
>>>           TOTAL:             654664
>>>
>>> Not sure if that helps or not.  Hotkernel shows the cpu in the
>>> kernel`acpi_cpu_c1  call most of the time, normal ?
>>>
>>> Dan
>>>
>>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel
>>>
>>> Sampling... Hit Ctrl-C to end.
>>> ^C
>>> FUNCTION                                                COUNT   PCNT
>>> zfs.ko`space_map_sync                                       1   0.0%
>>> kernel`syscall_thread_exit                                  1   0.0%
>>> kernel`should_yield                                         1   0.0%
>>> zfs.ko`lzjb_compress                                        1   0.0%
>>> zfs.ko`dbuf_rele                                            1   0.0%
>>> zfs.ko`dsl_pool_tempreserve_****space                           1   0.0%
>>>
>>> kernel`_bus_dmamap_load_buffer                              1   0.0%
>>> kernel`cpu_idleclock                                        1   0.0%
>>> kernel`_rw_runlock_cookie                                   1   0.0%
>>> kernel`pmap_enter                                           1   0.0%
>>> kernel`__lockmgr_args                                       1   0.0%
>>> zfs.ko`vdev_geom_io_start                                   1   0.0%
>>> kernel`ahci_ch_intr                                         1   0.0%
>>> kernel`__rw_rlock                                           1   0.0%
>>> kernel`VOP_UNLOCK_APV                                       1   0.0%
>>> zfs.ko`arc_buf_thaw                                         1   0.0%
>>> kernel`atomic_set_int                                       1   0.0%
>>> kernel`g_io_schedule_down                                   1   0.0%
>>> kernel`__mtx_lock_flags                                     1   0.0%
>>> kernel`_callout_stop_safe                                   1   0.0%
>>> kernel`sched_idletd                                         1   0.0%
>>> kernel`_sx_slock                                            1   0.0%
>>> vmm.ko`lapic_set_intr                                       1   0.0%
>>> kernel`cpusetobj_ffs                                        1   0.0%
>>> zfs.ko`dbuf_hold                                            1   0.0%
>>> vmm.ko`vlapic_intr_accepted                                 1   0.0%
>>> zfs.ko`sa_attr_op                                           1   0.0%
>>> zfs.ko`dmu_buf_hold_array_by_****dnode                          1   0.0%
>>>
>>> zfs.ko`taskq_run_safe                                       1   0.0%
>>> kernel`vm_reserv_free_page                                  1   0.0%
>>> kernel`_sx_sunlock                                          1   0.0%
>>> kernel`vn_rlimit_fsize                                      1   0.0%
>>> zfs.ko`vdev_queue_io_to_issue                               1   0.0%
>>> kernel`vn_write                                             1   0.0%
>>> kernel`lock_init                                            1   0.0%
>>> vmm.ko`vmx_resume                                           1   0.0%
>>> zfs.ko`zio_gang_tree_free                                   1   0.0%
>>> zfs.ko`dbuf_rele_and_unlock                                 1   0.0%
>>> kernel`mnt_vnode_next_active                                1   0.0%
>>> kernel`vm_page_remove                                       1   0.0%
>>> zfs.ko`vdev_queue_deadline_****compare                          1   0.0%
>>>
>>> kernel`zone_free_item                                       1   0.0%
>>> vmm.ko`vmx_launch                                           1   0.0%
>>> kernel`m_pullup                                             1   0.0%
>>> kernel`pmap_change_wiring                                   1   0.0%
>>> kernel`sched_userret                                        1   0.0%
>>> zfs.ko`dbuf_write_done                                      1   0.0%
>>> zfs.ko`zio_execute                                          1   0.0%
>>> kernel`PHYS_TO_VM_PAGE                                      2   0.0%
>>> zfs.ko`avl_rotation                                         2   0.0%
>>> kernel`ehci_device_isoc_fs_****enter                            2   0.0%
>>>
>>> vmm.ko`vm_exitinfo                                          2   0.0%
>>> zfs.ko`zfs_freebsd_write                                    2   0.0%
>>> vmm.ko`vcpu_stats                                           2   0.0%
>>> kernel`atomic_add_long                                      2   0.0%
>>> vmm.ko`0xffffffff81                                         2   0.0%
>>> kernel`vm_reserv_alloc_page                                 2   0.0%
>>> kernel`pagezero                                             2   0.0%
>>> kernel`bzero                                                2   0.0%
>>> zfs.ko`fletcher_2_native                                    2   0.0%
>>> vmm.ko`vmx_getreg                                           2   0.0%
>>> vmm.ko`vm_guest_msrs                                        3   0.0%
>>> vmm.ko`vmx_setreg                                           3   0.0%
>>> vmm.ko`vm_get_register                                      3   0.0%
>>> kernel`sx_try_xlock_                                        3   0.0%
>>> kernel`vm_radix_lookup_le                                   3   0.0%
>>> vmm.ko`lapic_pending_intr                                   4   0.0%
>>> kernel`syscall_thread_enter                                 4   0.0%
>>> kernel`cpu_set_syscall_retval                               4   0.0%
>>> kernel`__mnt_vnode_next_all                                 4   0.0%
>>> kernel`vfs_stdsync                                          4   0.0%
>>> vmm.ko`vmm_fetch_instruction                                5   0.0%
>>> vmm.ko`vlapic_op_mem_read                                   6   0.0%
>>> kernel`_sx_xunlock                                          7   0.0%
>>> vmm.ko`vmm_emulate_instruction                              7   0.0%
>>> vmm.ko`vlapic_op_mem_write                                  7   0.0%
>>> kernel`__mtx_lock_spin_flags                                8   0.0%
>>> vmm.ko`lapic_timer_tick                                     8   0.0%
>>> zfs.ko`list_next                                            8   0.0%
>>> zfs.ko`l2arc_feed_thread                                    8   0.0%
>>> kernel`userret                                              9   0.0%
>>> vmm.ko`vm_gpa2hpa                                           9   0.0%
>>> vmm.ko`vmm_decode_instruction                              11   0.0%
>>> vmm.ko`ept_vmmmap_get                                      11   0.0%
>>> vmm.ko`vm_lapic                                            11   0.0%
>>> kernel`spinlock_enter                                      14   0.0%
>>> kernel`__mtx_lock_sleep                                    15   0.0%
>>> vmm.ko`vcpu_set_state                                      16   0.0%
>>> vmm.ko`vm_nmi_pending                                      16   0.0%
>>> kernel`malloc_type_zone_****allocated                          16   0.0%
>>>
>>> kernel`devfs_ioctl_f                                       16   0.0%
>>> kernel`pmap_kextract                                       17   0.0%
>>> kernel`sys_ioctl                                           18   0.0%
>>> kernel`malloc                                              21   0.0%
>>> kernel`cpu_fetch_syscall_args                              24   0.0%
>>> kernel`uma_zfree_arg                                       28   0.0%
>>> vmm.ko`vlapic_pending_intr                                 29   0.0%
>>> kernel`trap                                                30   0.0%
>>> kernel`hpet_get_timecount                                  31   0.0%
>>> dtrace.ko`dtrace_trap                                      35   0.0%
>>> vmm.ko`vlapic_update_ppr                                   39   0.0%
>>> vmm.ko`vmmdev_ioctl                                        40   0.0%
>>> kernel`copyout                                             43   0.0%
>>> kernel`critical_enter                                      44   0.0%
>>> kernel`copyin                                              51   0.1%
>>> vmm.ko`vmcs_read                                           53   0.1%
>>> kernel`fpusave                                             61   0.1%
>>> kernel`uma_zalloc_arg                                      62   0.1%
>>> kernel`devvn_refthread                                     69   0.1%
>>> kernel`atomic_load_acq_int                                 70   0.1%
>>> kernel`fpurestore                                          70   0.1%
>>> vmm.ko`vlapic_timer_tick                                   75   0.1%
>>> kernel`amd64_syscall                                       77   0.1%
>>> kernel`bcopy                                               91   0.1%
>>> kernel`fpudna                                             103   0.1%
>>> kernel`free                                               103   0.1%
>>> kernel`critical_exit                                      105   0.1%
>>> kernel`dev_relthread                                      108   0.1%
>>> kernel`spinlock_exit                                      119   0.1%
>>> kernel`kern_ioctl                                         138   0.1%
>>> kernel`fpuexit                                            242   0.3%
>>> kernel`0xffffffff80                                       306   0.3%
>>> vmm.ko`restore_host_msrs                                  398   0.4%
>>> vmm.ko`vm_run                                             428   0.4%
>>> vmm.ko`restore_guest_msrs                                 433   0.5%
>>> kernel`cpu_idle                                           906   1.0%
>>> vmm.ko`vmx_run                                           9266   9.7%
>>> kernel`acpi_cpu_c1                                      81134  85.2%
>>>
>>> Hope this helps, let me know if you have any ideas on what to try next.
>>>
>>>
>>>  Could you try the following command to get vcpu stats?
>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0
>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1
>>
>> best
>> Neel
>>
>>
>>  Dan
>>>
>>>
>>>
>>>
>>> On Fri, 29 Mar 2013, Dan Mack wrote:
>>>
>>>  On Thu, 28 Mar 2013, Peter Grehan wrote:
>>>
>>>>
>>>>  Hi Dan,
>>>>
>>>>>
>>>>>  Any ideas?
>>>>>
>>>>>>
>>>>>>
>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6
>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest.
>>>>>
>>>>>
>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM.
>>>>
>>>>  The issue you are seeing looks a bit like what happens when systems are
>>>>
>>>>> heavily oversubscribed and vCPUs can't get enough run time.
>>>>>
>>>>>
>>>> I was running in on an idle system; only bhyve was running.
>>>>
>>>>
>>>>  How much spare RAM is on the host when you started the vm ? Also, is
>>>>
>>>>> there much running on the host when the guest is started ?
>>>>>
>>>>>
>>>> 64GB
>>>>
>>>> Here's some more information (sorry for the formatting):
>>>>
>>>> Case 1:  single vCPU via:
>>>> root 64112    1.3  0.0 4219232 25588  1  D+    2:09PM     0:02.47
>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>
>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB
>>>> free RAM, nothing else clocking CPU cycles):
>>>>
>>>> root@olive:~ # top -bH
>>>> last pid: 64148;  load averages:  0.03,  0.22,  0.22  up 0+16:29:31
>>>> 14:11:51
>>>> 24 processes:  1 running, 23 sleeping
>>>>
>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free
>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other
>>>> Swap:
>>>>
>>>>
>>>>  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>>>> 64112 root        20    0  4120M 25980K vmidle  2   0:03  0.00%
>>>> bhyve{coco vcpu 0}
>>>>  925 root        20    0 14296K  2044K select 11   0:00  0.00% syslogd
>>>> 2695 root        20    0 81452K  6952K select  2   0:00  0.00% sshd
>>>> 2697 root        20    0 20224K  4632K ttyin   1   0:00  0.00% csh
>>>> 64112 root        52    0  4120M 25980K kqread  9   0:00  0.00%
>>>> bhyve{coco mevent}
>>>>
>>>>
>>>> So, case #1 all is normal.  I was able to build a kernel with very good
>>>> performance.
>>>>
>>>> The boot sequence looks like the following:
>>>>
>>>> root@olive:/vms/coco # ./vmrun.sh coco
>>>> Launching virtual machine "coco" ...
>>>> Consoles: userboot
>>>>
>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>> (r...@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>
>>>> /
>>>>  ______               ____   _____ _____
>>>> |  ____|             |  _ \ / ____|  __ \
>>>> | |___ _ __ ___  ___ | |_) | (___ | |  | |
>>>> |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
>>>> | |   | | |  __/  __/| |_) |____) | |__| |
>>>> | |   | | |    |    ||     |      |      |
>>>> |_|   |_|  \___|\___||____/|_____/|_____/    ```
>>>>  `
>>>>                                             s` `.....---.......--.```
>>>> -/
>>>> +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`
>>>>  +.
>>>> |                                         |  yo`:.            :o
>>>>  `+-
>>>> |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
>>>> |  2. Boot [S]ingle User                  |  .-
>>>>  ::/sy+:.
>>>> |  3. [Esc]ape to loader prompt           |  /                     `--
>>>>  /
>>>> |  4. Reboot                              | `:
>>>>  :`
>>>> |                                         | `:
>>>>  :`
>>>> |  Options:                               |  /
>>>>  /
>>>> |  5. Configure Boot [O]ptions...         |  .-
>>>>  -.
>>>> |                                         |   --                      -.
>>>> |                                         |    `:`                  `:`
>>>> |                                         |      .--             `--.
>>>> |                                         |         .---.....----.
>>>> +-----------------------------****------------+
>>>>
>>>>
>>>>
>>>> Booting...
>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>>         The Regents of the University of California. All rights
>>>> reserved.
>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>    r...@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>
>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU)
>>>>  Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d
>>>>  Stepping = 7
>>>>  Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>  Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>> TSCDLT,**AESNI,HV>
>>>>  AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>
>>>>  AMD Features2=0x1<LAHF>
>>>>  TSC: P-state invariant
>>>> real memory  = 5368709120 (5120 MB)
>>>> avail memory = 3957534720 (3774 MB)
>>>> Event timer "LAPIC" quality 400
>>>> ACPI APIC Table: <BHYVE  BVMADT  >
>>>> random device not loaded; using insecure entropy
>>>> ioapic0: Changing APIC ID to 1
>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>> kbd0 at kbdmux0
>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>> pci0: <ACPI PCI bus> on pcib0
>>>> pcib0: no PRT entry for 0.31.INTA
>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>> pci0
>>>> uart2: console (9600,n,8,1)
>>>> Timecounters tick every 10.000 msec
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000
>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>
>>>> Setting hostid: 0x24bfd025.
>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>> Starting file system checks:
>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7%
>>>> fragmentation)
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> Mounting local file systems:.
>>>> Writing entropy file:.
>>>> Setting hostname: cocopuff.example.com.
>>>> vtnet0: link state changed to UP
>>>> Starting Network: lo0 vtnet0.
>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>         options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>
>>>>         inet6 ::1 prefixlen 128
>>>>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>         inet 127.0.0.1 netmask 0xff000000
>>>>         nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>> MULTICAST>
>>>> metric 0 mtu 1500
>>>>         options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>
>>>>         ether 00:a0:98:ac:0d:df
>>>>         inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>         inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>         nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>
>>>>         media: Ethernet 1000baseT <full-duplex>
>>>>         status: active
>>>> Starting devd.
>>>> add net default: gateway 172.22.22.252
>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>> add net ::0.0.0.0: gateway ::1
>>>> add net fe80::: gateway ::1
>>>> add net ff02::: gateway ::1
>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>> Creating and/or trimming log files.
>>>> Starting syslogd.
>>>> Clearing /tmp (X related).
>>>> Updating motd:.
>>>> Performing sanity check on sshd configuration.
>>>> Starting sshd.
>>>> Starting cron.
>>>> Starting background file system checks in 60 seconds.
>>>>
>>>> Fri Mar 29 14:09:20 CDT 2013
>>>>
>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>
>>>> login:
>>>>
>>>>
>>>> ====== Next, boot the same image with two (2) vCPU:
>>>>
>>>> Command run by vmrun.sh:
>>>>
>>>> root 64163  182.5  0.1 4219360 48968  1  R+    2:16PM     1:10.42
>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>
>>>> Top shows it running out of control:
>>>>
>>>> root@olive:~ # top -bH
>>>> last pid: 64191;  load averages:  1.45,  0.52,  0.31  up 0+16:35:12
>>>> 14:17:32
>>>> 25 processes:  3 running, 22 sleeping
>>>>
>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free
>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other
>>>> Swap:
>>>>
>>>>
>>>>  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>>>> 64163 root       102    0  4120M   120M CPU2    2   1:01 100.00%
>>>> bhyve{coco vcpu 1}
>>>> 64163 root       102    0  4120M   120M CPU8    8   1:03 94.48%
>>>> bhyve{coco vcpu 0}
>>>> 15751 root        20    0 81452K  7092K select  5   0:00  0.00% sshd
>>>>  807 root        20    0 13112K  4812K select  5   0:00  0.00% devd
>>>>  925 root        20    0 14296K  2044K select  9   0:00  0.00% syslogd
>>>> 1031 root        20    0 16380K  2400K nanslp 10   0:00  0.00% cron
>>>> 2695 root        20    0 81452K  6952K select  2   0:00  0.00% sshd
>>>> 2697 root        20    0 20224K  4632K ttyin   1   0:00  0.00% csh
>>>> 64163 root        52    0  4120M   120M kqread  5   0:00  0.00%
>>>> bhyve{coco mevent}
>>>> 15753 root        20    0 20224K  4580K pause   1   0:00  0.00% csh
>>>>
>>>> Even though, it's just sitting at the login prompt:
>>>>
>>>> root@olive:/vms/coco # ./vmrun.sh coco
>>>> Launching virtual machine "coco" ...
>>>> Consoles: userboot
>>>>
>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>> (r...@olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>
>>>> /
>>>>  ______               ____   _____ _____
>>>> |  ____|             |  _ \ / ____|  __ \
>>>> | |___ _ __ ___  ___ | |_) | (___ | |  | |
>>>> |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
>>>> | |   | | |  __/  __/| |_) |____) | |__| |
>>>> | |   | | |    |    ||     |      |      |
>>>> |_|   |_|  \___|\___||____/|_____/|_____/    ```
>>>>  `
>>>>                                             s` `.....---.......--.```
>>>> -/
>>>> +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`
>>>>  +.
>>>> |                                         |  yo`:.            :o
>>>>  `+-
>>>> |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
>>>> |  2. Boot [S]ingle User                  |  .-
>>>>  ::/sy+:.
>>>> |  3. [Esc]ape to loader prompt           |  /                     `--
>>>>  /
>>>> |  4. Reboot                              | `:
>>>>  :`
>>>> |                                         | `:
>>>>  :`
>>>> |  Options:                               |  /
>>>>  /
>>>> |  5. Configure Boot [O]ptions...         |  .-
>>>>  -.
>>>> |                                         |   --                      -.
>>>> |                                         |    `:`                  `:`
>>>> |                                         |      .--             `--.
>>>> |                                         |         .---.....----.
>>>> +-----------------------------****------------+
>>>>
>>>>
>>>>
>>>> Booting...
>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>>         The Regents of the University of California. All rights
>>>> reserved.
>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>    r...@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>
>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU)
>>>>  Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d
>>>>  Stepping = 7
>>>>  Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>  Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>> TSCDLT,**AESNI,HV>
>>>>  AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>
>>>>  AMD Features2=0x1<LAHF>
>>>>  TSC: P-state invariant
>>>> real memory  = 5368709120 (5120 MB)
>>>> avail memory = 3957534720 (3774 MB)
>>>> Event timer "LAPIC" quality 400
>>>> ACPI APIC Table: <BHYVE  BVMADT  >
>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>>>> FreeBSD/SMP: 2 package(s) x 1 core(s)
>>>> cpu0 (BSP): APIC ID:  0
>>>> cpu1 (AP): APIC ID:  1
>>>> random device not loaded; using insecure entropy
>>>> ioapic0: Changing APIC ID to 2
>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>> kbd0 at kbdmux0
>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>> pci0: <ACPI PCI bus> on pcib0
>>>> pcib0: no PRT entry for 0.31.INTA
>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>> pci0
>>>> uart2: console (9600,n,8,1)
>>>> Timecounters tick every 10.000 msec
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> SMP: AP CPU #1 Launched!
>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>
>>>> Setting hostid: 0x24bfd025.
>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>> Starting file system checks:
>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7%
>>>> fragmentation)
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>> Mounting local file systems:.
>>>> Writing entropy file:.
>>>> Setting hostname: cocopuff.example.com.
>>>> vtnet0: link state changed to UP
>>>> Starting Network: lo0 vtnet0.
>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>         options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>
>>>>         inet6 ::1 prefixlen 128
>>>>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>         inet 127.0.0.1 netmask 0xff000000
>>>>         nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>> MULTICAST>
>>>> metric 0 mtu 1500
>>>>         options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>
>>>>         ether 00:a0:98:ac:0d:df
>>>>         inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>         inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>         nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>
>>>>         media: Ethernet 1000baseT <full-duplex>
>>>>         status: active
>>>> Starting devd.
>>>> add net default: gateway 172.22.22.252
>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>> add net ::0.0.0.0: gateway ::1
>>>> add net fe80::: gateway ::1
>>>> add net ff02::: gateway ::1
>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>> Creating and/or trimming log files.
>>>> Starting syslogd.
>>>> syslogd: timed out waiting for child
>>>> /etc/rc: WARNING: failed to start syslogd
>>>> Clearing /tmp (X related).
>>>> Updating motd:.
>>>> Performing sanity check on sshd configuration.
>>>> Starting sshd.
>>>> Starting cron.
>>>> Starting background file system checks in 60 seconds.
>>>>
>>>> Thu Apr 18 02:06:43 CDT 2013
>>>>
>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>
>>>> login:
>>>>
>>>>
>>>> ========== end of boot sequence
>>>>
>>>> So, something is broken with SMP on the guest, I'd be happy to help just
>>>> let me know.
>>>>
>>>> Dan
>>>>
>>>> ______________________________****_________________
>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization@freebsd.org>>mailing
>>>> list
>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****
>>>> virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>>> >
>>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>>> unsubscr...@freebsd.org <freebsd-virtualization-**
>>>> unsubscr...@freebsd.org<freebsd-virtualization-unsubscr...@freebsd.org>
>>>> >"
>>>>
>>>>  ______________________________****_________________
>>>>
>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization@freebsd.org>>mailing
>>> list
>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>> >
>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>> unsubscr...@freebsd.org <freebsd-virtualization-**
>>> unsubscr...@freebsd.org <freebsd-virtualization-unsubscr...@freebsd.org>
>>> >"
>>>
>>>
>>
_______________________________________________
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Reply via email to