Hi Robert, there are two other KVMs running on the same host, but there comes almost no load from them. CPU cap ist not set.
If the KVM does nothing, most of the qemu-processes are near 100% SLP rate (as expected). LAT is very low near 0,0 / 0,1 %. If i fire up prime things change: SLP is near 0% and LAT is quite high near 100%. I copied some example data for you: No load: ### PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 99285 root 0,3 0,1 0,0 0,0 0,0 0,0 100 0,1 862 6 6K 0 qemu-system-/1 99285 root 0,0 0,2 0,0 0,0 0,0 0,0 100 0,0 125 0 0 0 qemu-system-/4 99285 root 0,0 0,1 0,0 0,0 0,0 0,0 100 0,0 676 0 1K 676 qemu-system-/19 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 22 0 0 0 qemu-system-/11 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 26 0 0 0 qemu-system-/10 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 18 0 0 0 qemu-system-/5 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 21 0 42 0 qemu-system-/3 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 29 0 2 0 qemu-system-/9 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 16 0 0 0 qemu-system-/18 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 16 0 42 0 qemu-system-/15 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 17 1 1 0 qemu-system-/7 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 15 0 0 0 qemu-system-/16 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 7 1 1 0 qemu-system-/14 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 11 0 0 0 qemu-system-/12 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 12 0 0 0 qemu-system-/8 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 3 0 0 0 qemu-system-/13 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 2 0 0 0 qemu-system-/17 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/19883 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 1 0 0 0 qemu-system-/6 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 0 0 0 0 qemu-system-/2 ### prime load: ### PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 99285 root 0,0 100 0,0 0,0 0,0 0,0 0,0 0,0 0 0 0 0 qemu-system-/12 99285 root 0,0 100 0,0 0,0 0,0 0,0 0,0 0,0 0 0 0 0 qemu-system-/8 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/19887 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/19 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/18 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/17 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/16 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/15 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/14 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/13 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/11 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/10 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/9 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/7 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/6 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/5 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/4 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/3 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 0 0 0 0 qemu-system-/2 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/1 PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 99285 root 0,0 100 0,0 0,0 0,0 0,0 0,0 0,0 0 0 0 0 qemu-system-/8 99285 root 0,0 3,8 0,0 0,0 0,0 0,0 0,0 96 3 24 26 0 qemu-system-/14 99285 root 0,0 3,6 0,0 0,0 0,0 0,0 0,0 96 15 24 41 0 qemu-system-/11 99285 root 0,0 1,9 0,0 0,0 0,0 0,0 0,0 98 0 17 15 0 qemu-system-/4 99285 root 0,0 1,0 0,0 0,0 0,0 0,0 0,0 99 1 31 33 0 qemu-system-/10 99285 root 0,0 0,9 0,0 0,0 0,0 0,1 0,0 99 7 31 34 0 qemu-system-/5 99285 root 0,0 0,9 0,0 0,0 0,0 0,0 0,0 99 1 24 24 0 qemu-system-/16 99285 root 0,0 0,9 0,0 0,0 0,0 0,0 0,0 99 0 25 28 0 qemu-system-/13 99285 root 0,0 0,7 0,0 0,0 0,0 0,1 0,0 99 6 19 23 0 qemu-system-/9 99285 root 0,0 0,7 0,0 0,0 0,0 0,0 0,0 99 2 24 25 0 qemu-system-/17 99285 root 0,0 0,7 0,0 0,0 0,0 0,0 0,0 99 3 33 38 0 qemu-system-/18 99285 root 0,1 0,0 0,0 0,0 0,0 0,0 31 68 31 16 433 0 qemu-system-/1 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 2 1 0 qemu-system-/6 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 20 80 45 19 90 45 qemu-system-/19 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 1 1 0 qemu-system-/12 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/15 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/7 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/3 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 0 0 0 0 qemu-system-/2 PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 99285 root 0,0 100 0,0 0,0 0,0 0,0 0,0 0,0 0 0 0 0 qemu-system-/6 99285 root 0,0 0,4 0,0 0,0 0,0 0,0 0,0 100 0 1 1 0 qemu-system-/13 99285 root 0,0 0,4 0,0 0,0 0,0 0,0 0,0 100 0 1 1 0 qemu-system-/9 99285 root 0,0 0,4 0,0 0,0 0,0 0,0 0,0 100 0 1 2 0 qemu-system-/5 99285 root 0,0 0,2 0,0 0,0 0,0 0,0 0,0 100 0 1 2 0 qemu-system-/17 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 1 2 52 0 qemu-system-/1 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 1 1 2 1 qemu-system-/19 99285 root 0,0 0,0 0,0 0,0 0,0 66 0,0 34 1 0 1 0 qemu-system-/14 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 1 1 1 0 qemu-system-/10 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/19890 99285 root 0,0 0,0 0,0 0,0 0,0 100 0,0 0,0 0 0 0 0 qemu-system-/19889 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/18 99285 root 0,0 0,0 0,0 0,0 0,0 66 0,0 34 1 0 1 0 qemu-system-/16 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/15 99285 root 0,0 0,0 0,0 0,0 0,0 33 0,0 67 0 1 0 0 qemu-system-/12 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/11 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/8 99285 root 0,0 0,0 0,0 0,0 0,0 66 0,0 34 1 0 1 0 qemu-system-/7 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/4 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 0,0 100 0 0 0 0 qemu-system-/3 99285 root 0,0 0,0 0,0 0,0 0,0 0,0 100 0,0 0 0 0 0 qemu-system-/2 ### Greets Kilian ________________________________________ Von: Robert Mustacchi <[email protected]> Gesendet: Donnerstag, 11. August 2016 03:15 An: [email protected] Betreff: Re: [smartos-discuss] KVM: BUG: soft lockup - CPU#2 stuck for 30s On 8/8/16 2:23 , Kilian Ries wrote: > Hello, > > > running different versions of SmartOS (up to the latest release > 20160803T101331Z) i have noticed a strange behaviour: > > > When SmartOS runs on a host with one CPU and 4 cores (4 real cores, 8 HT > cores) i can set as many vCPUs for my KVM as i want. If i set it up to for > example 12 vCPUs and run prime inside my VM, it takes about one minute and > i'm seeing many messages like this: > > > > ### > > > [Do Aug 4 17:22:37 2016] BUG: soft lockup - CPU#6 stuck for 27s! > [mprime:2796] > > [Do Aug 4 17:22:37 2016] Modules linked in: > > [Do Aug 4 17:22:37 2016] ppdev > > [Do Aug 4 17:22:37 2016] parport_pc > > [Do Aug 4 17:22:37 2016] sg > > [Do Aug 4 17:22:37 2016] parport > > [Do Aug 4 17:22:37 2016] pcspkr > > [Do Aug 4 17:22:37 2016] i2c_piix4 > > [Do Aug 4 17:22:37 2016] ip_tables > > [Do Aug 4 17:22:37 2016] xfs > > [Do Aug 4 17:22:37 2016] libcrc32c > > [Do Aug 4 17:22:37 2016] sr_mod > > [Do Aug 4 17:22:37 2016] cdrom > > [Do Aug 4 17:22:37 2016] ata_generic > > [Do Aug 4 17:22:37 2016] pata_acpi > > [Do Aug 4 17:22:37 2016] bochs_drm > > [Do Aug 4 17:22:37 2016] syscopyarea > > [Do Aug 4 17:22:37 2016] sysfillrect > > [Do Aug 4 17:22:37 2016] sysimgblt > > [Do Aug 4 17:22:37 2016] drm_kms_helper > > [Do Aug 4 17:22:37 2016] ttm > > [Do Aug 4 17:22:37 2016] drm > > [Do Aug 4 17:22:37 2016] ata_piix > > [Do Aug 4 17:22:37 2016] virtio_net > > [Do Aug 4 17:22:37 2016] virtio_blk > > [Do Aug 4 17:22:37 2016] virtio_pci > > [Do Aug 4 17:22:37 2016] i2c_core > > [Do Aug 4 17:22:37 2016] virtio_ring > > [Do Aug 4 17:22:37 2016] libata > > [Do Aug 4 17:22:37 2016] floppy > > [Do Aug 4 17:22:37 2016] serio_raw > > [Do Aug 4 17:22:37 2016] virtio > > [Do Aug 4 17:22:37 2016] dm_mirror > > [Do Aug 4 17:22:37 2016] dm_region_hash > > [Do Aug 4 17:22:37 2016] dm_log > > [Do Aug 4 17:22:37 2016] dm_mod > > > [Do Aug 4 17:22:37 2016] CPU: 6 PID: 2796 Comm: mprime Tainted: G > L ------------ 3.10.0-327.22.2.el7.x86_64 #1 > > [Do Aug 4 17:22:37 2016] Hardware name: Joyent SmartDC HVM, BIOS Bochs > 01/01/2007 > > [Do Aug 4 17:22:37 2016] task: ffff8803efd26780 ti: ffff8803eed04000 > task.ti: ffff8803eed04000 > > [Do Aug 4 17:22:37 2016] RIP: 0033:[<000000000164dd78>] > > [Do Aug 4 17:22:37 2016] [<000000000164dd78>] 0x164dd77 > > [Do Aug 4 17:22:37 2016] RSP: 002b:00007f6f917f8710 EFLAGS: 00000202 > > [Do Aug 4 17:22:37 2016] RAX: 000000004000000d RBX: 000000000000fe2e RCX: > 00007f6f4afb4b00 > > [Do Aug 4 17:22:37 2016] RDX: 0000000000000000 RSI: 00007f6f4afaa480 RDI: > 00007f6f9823c400 > > [Do Aug 4 17:22:37 2016] RBP: 00007f6f9823c400 R08: 00007f6f4afa9780 R09: > 0000000000000000 > > [Do Aug 4 17:22:37 2016] R10: 00007f6f917f8840 R11: 00007f6f80001000 R12: > 00007f6f9823c800 > > [Do Aug 4 17:22:37 2016] R13: 00007f6f9823c500 R14: ffffffff8163c831 R15: > ffff8803eed07f70 > > [Do Aug 4 17:22:37 2016] FS: 00007f6f917fa700(0000) > GS:ffff880407cc0000(0000) knlGS:0000000000000000 > > [Do Aug 4 17:22:37 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [Do Aug 4 17:22:37 2016] CR2: 00007f9ebbb5c000 CR3: 00000003efd1d000 CR4: > 00000000000006e0 > > [Do Aug 4 17:22:37 2016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > [Do Aug 4 17:22:37 2016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > > [Do Aug 4 17:22:40 2016] BUG: soft lockup - CPU#2 stuck for 30s! > [mprime:2792] > > [Do Aug 4 17:22:40 2016] Modules linked in: > > [Do Aug 4 17:22:40 2016] ppdev > > [Do Aug 4 17:22:40 2016] parport_pc > > [Do Aug 4 17:22:40 2016] sg > > [Do Aug 4 17:22:40 2016] parport > > [Do Aug 4 17:22:40 2016] pcspkr > > [Do Aug 4 17:22:40 2016] i2c_piix4 > > [Do Aug 4 17:22:40 2016] ip_tables > > [Do Aug 4 17:22:40 2016] xfs > > [Do Aug 4 17:22:40 2016] libcrc32c > > [Do Aug 4 17:22:40 2016] sr_mod > > [Do Aug 4 17:22:40 2016] cdrom > > [Do Aug 4 17:22:40 2016] ata_generic > > [Do Aug 4 17:22:40 2016] pata_acpi > > [Do Aug 4 17:22:40 2016] bochs_drm > > [Do Aug 4 17:22:40 2016] syscopyarea > > [Do Aug 4 17:22:40 2016] sysfillrect > > [Do Aug 4 17:22:40 2016] sysimgblt > > [Do Aug 4 17:22:40 2016] drm_kms_helper > > [Do Aug 4 17:22:40 2016] ttm > > [Do Aug 4 17:22:40 2016] drm > > [Do Aug 4 17:22:40 2016] ata_piix > > [Do Aug 4 17:22:40 2016] virtio_net > > [Do Aug 4 17:22:40 2016] virtio_blk > > [Do Aug 4 17:22:40 2016] virtio_pci > > [Do Aug 4 17:22:40 2016] i2c_core > > [Do Aug 4 17:22:40 2016] virtio_ring > > [Do Aug 4 17:22:40 2016] libata > > [Do Aug 4 17:22:40 2016] floppy > > [Do Aug 4 17:22:40 2016] serio_raw > > [Do Aug 4 17:22:40 2016] virtio > > [Do Aug 4 17:22:40 2016] dm_mirror > > [Do Aug 4 17:22:40 2016] dm_region_hash > > [Do Aug 4 17:22:40 2016] dm_log > > [Do Aug 4 17:22:40 2016] dm_mod > > > [Do Aug 4 17:22:40 2016] CPU: 2 PID: 2792 Comm: mprime Tainted: G > L ------------ 3.10.0-327.22.2.el7.x86_64 #1 > > [Do Aug 4 17:22:40 2016] Hardware name: Joyent SmartDC HVM, BIOS Bochs > 01/01/2007 > > [Do Aug 4 17:22:40 2016] task: ffff8803efd23980 ti: ffff8803ef6d8000 > task.ti: ffff8803ef6d8000 > > [Do Aug 4 17:22:40 2016] RIP: 0033:[<00007f6f997d7995>] > > [Do Aug 4 17:22:40 2016] [<00007f6f997d7995>] 0x7f6f997d7994 > > [Do Aug 4 17:22:40 2016] RSP: 002b:00007f6f937fc708 EFLAGS: 00000202 > > [Do Aug 4 17:22:40 2016] RAX: 414593d9a0beeb7d RBX: 000000000000fe2e RCX: > 40cdbb5e7a93de49 > > [Do Aug 4 17:22:40 2016] RDX: 000000000008c1d8 RSI: 00007f6edd0e1440 RDI: > 00007f6edd5f04b8 > > [Do Aug 4 17:22:40 2016] RBP: 00007f6edd0332a0 R08: 0000000000000000 R09: > 0000000000000099 > > [Do Aug 4 17:22:40 2016] R10: 0000000000000001 R11: 00007f6f50001000 R12: > 00007f6f7f2f8f80 > > [Do Aug 4 17:22:40 2016] R13: 00007f6f50002640 R14: ffffffff8163c831 R15: > ffff8803ef6dbf70 > > [Do Aug 4 17:22:40 2016] FS: 00007f6f937fe700(0000) > GS:ffff880407c40000(0000) knlGS:0000000000000000 > > [Do Aug 4 17:22:40 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [Do Aug 4 17:22:40 2016] CR2: 0000000002541eb0 CR3: 00000003efd1d000 CR4: > 00000000000006e0 > > [Do Aug 4 17:22:40 2016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > [Do Aug 4 17:22:40 2016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > > [Do Aug 4 17:22:40 2016] libata > > [Do Aug 4 17:22:40 2016] floppy serio_raw virtio dm_mirror dm_region_hash > dm_log dm_mod > > [Do Aug 4 17:22:40 2016] CPU: 9 PID: 2799 Comm: mprime Tainted: G > L ------------ 3.10.0-327.22.2.el7.x86_64 #1 > > [Do Aug 4 17:22:40 2016] Hardware name: Joyent SmartDC HVM, BIOS Bochs > 01/01/2007 > > [Do Aug 4 17:22:40 2016] task: ffff8803efd25c00 ti: ffff8803efcc0000 > task.ti: ffff8803efcc0000 > > [Do Aug 4 17:22:40 2016] RIP: 0033:[<000000000164ebb0>] > [<000000000164ebb0>] 0x164ebaf > > [Do Aug 4 17:22:40 2016] RSP: 002b:00007f6f8b7fc710 EFLAGS: 00000202 > > [Do Aug 4 17:22:40 2016] RAX: 0000000000000010 RBX: 000000000000fe2e RCX: > 00007f6ee35d6280 > > [Do Aug 4 17:22:40 2016] RDX: 0000000000000000 RSI: 00007f6ee35c8e40 RDI: > 00007f6f7f0aa440 > > [Do Aug 4 17:22:40 2016] RBP: 00007f6f7f0aa800 R08: 00007f6ee35c8e00 R09: > 0000000000000000 > > [Do Aug 4 17:22:40 2016] R10: 00007f6f8b7fc840 R11: 00007f6f5c001000 R12: > 00007f6f7f0aa800 > > [Do Aug 4 17:22:40 2016] R13: 00007f6f7f0aa500 R14: ffffffff8163c831 R15: > ffff8803efcc3f70 > > [Do Aug 4 17:22:40 2016] FS: 00007f6f8b7fe700(0000) > GS:ffff880407d20000(0000) knlGS:0000000000000000 > > [Do Aug 4 17:22:40 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [Do Aug 4 17:22:40 2016] CR2: 00007fe925f5f292 CR3: 00000003efd1d000 CR4: > 00000000000006e0 > > [Do Aug 4 17:22:40 2016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > [Do Aug 4 17:22:40 2016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > > ### > > > > After some time the KVM is complete unusable, SSH login doesn't work and i > have no other solution than to force poweroff (vmadm stop -f UUID). If i set > the KVM to only 4 vCPUs everything runs fine and prime runs for > 30 minutes > without problems. > > > As i know from other virtualization software (VMware, ...) you are not > allowed to setup more vCPUs / VM than the host itself has. But for example in > Proxmox, if you are using QEMU Virtual CPUs (as SmartOS does) you are allowed > to set as many vCPUs as you want. > > > So is this a bug in SmartOS and you shouldn't be able to set more vCPUs / KVM > as the host itself has or when not, why isn't SmartOS capable of handling the > high load correctly (KVM shouldn't freeze)? Hi Killian, So at the end of the day, SmartOS is not going to stop you from over-provisioning on pretty much any axis. So in theory we'll be trying to time-share this CPU against all the others. So, a couple of questions to better understand the situation. Is this the only zone running on the box? Is there a CPU cap set? If you look at the QEMU process with prstat -mL at say a per-second rate, what do you end up seeing, if anything there? Do you see some threads (which represent CPUs) spending all of their time in a certain state or is something else going on? Is there a lot of LAT there for some reason? Robert ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
