Re: Q: Stopped VM still using host cpu CPU ?
On 10/21/2009 10:29 PM, Daniel Schwager wrote:
opreport -l --symbols | less
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples %image name app name
symbol name
418814 98.5250 no-vmlinux no-vmlinux (no
symbols)
1228 0.2889 qemu-system-x86_64 qemu-system-x86_64
main_loop_wait
888 0.2089 libpthread-2.8.solibpthread-2.8.so
__read_nocancel
...
Sorry, I gave you the wrong instructions. Since most of the time is
spent in vmlinux, we want to trace vmlinux, not qemu.
Three ways to see what is happening:
- set up vmlinux (you'll need to install the correct kernel-debuginfo
package to get this).
- strace -p `pgrep qemu` and see what syscalls take place
- run 'top' and look at the wchan field ('f' 'y' will show it if it is
hidden)
Probably easiest in reverse order as listed.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
Hi Avi, so, setup with opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start Use 'opreport -l'. Make sure your qemu isn't stripped. All VM's are in paused state: top - 22:08:15 up 2 days, 12:18, 8 users, load average: 0.12, 0.19, 0.14 Tasks: 185 total, 1 running, 182 sleeping, 2 stopped, 0 zombie Cpu(s): 0.5%us, 1.7%sy, 0.0%ni, 97.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8196468k total, 4166468k used, 403k free,76044k buffers Swap: 39780320k total,70232k used, 39710088k free, 475876k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 9587 root 20 0 1145m 1.0g 1944 S 6.6 13.1 0:28.78 qemu-system-x86 9525 root 20 0 1143m 1.0g 1900 S 6.3 13.1 0:27.26 qemu-system-x86 9305 root 20 0 1143m 1.0g 1900 S 5.0 13.1 0:22.86 qemu-system-x86 opreport -l --symbols | less CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples %image name app name symbol name 418814 98.5250 no-vmlinux no-vmlinux (no symbols) 1228 0.2889 qemu-system-x86_64 qemu-system-x86_64 main_loop_wait 888 0.2089 libpthread-2.8.solibpthread-2.8.so __read_nocancel ... opreport -l /opt/kvm-86-vnc-patch.oprofile/usr/bin/qemu-system-x86_64 --symbols | less CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples %image name symbol name 1036 52.2441 qemu-system-x86_64 main_loop_wait 155 7.8164 qemu-system-x86_64 dynticks_rearm_timer 854.2864 [vdso] (tgid:9525 range:0x7fff565fe000-0x7fff565ff000) (no symbols) 773.8830 qemu-system-x86_64 qemu_get_clock 633.1770 qemu-system-x86_64 .plt 592.9753 qemu-system-x86_64 host_alarm_handler 572.8744 qemu-system-x86_64 qemu_shutdown_requested 512.5719 [vdso] (tgid:9305 range:0x7fff3d9ff000-0x7fff3da0) (no symbols) 492.4710 qemu-system-x86_64 sigfd_handler 472.3701 qemu-system-x86_64 tap_can_send 412.0676 [vdso] (tgid:9587 range:0x7abfe000-0x7abff000) (no symbols) 381.9163 qemu-system-x86_64 kvm_mutex_lock 361.8154 qemu-system-x86_64 kvm_main_loop 311.5633 qemu-system-x86_64 get_clock 291.4624 qemu-system-x86_64 qemu_bh_poll 241.2103 qemu-system-x86_64 io_thread_wakeup 231.1599 qemu-system-x86_64 qemu_event_read 190.9581 qemu-system-x86_64 qemu_powerdown_requested 180.9077 qemu-system-x86_64 kvm_mutex_unlock 120.6051 qemu-system-x86_64 qemu_kvm_notify_work 120.6051 qemu-system-x86_64 qemu_reset_requested 110.5547 qemu-system-x86_64 e1000_can_receive 6 0.3026 qemu-system-x86_64 slirp_is_inited 3 0.1513 qemu-system-x86_64 qemu_notify_event 1 0.0504 qemu-system-x86_64 qemu_set_fd_handler2 I do not understand, why only ~1036 kvm-samples produces a load of (in sum) 18% ... I'm not able to convert vmlinuz to vmlinux (for profiling the kernel ...) on my old FC9 - sorry. Because I cannot update my old fc9, I will move to CentOS later - and hopefully get the kernel profiling running. May these oprofile-information will help you to track down the problem - if not, I try to came back soon with kernel-profiling information on CentOS. regards Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
so, setup with opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start Use 'opreport -l'. Make sure your qemu isn't stripped. ok ... will do Try to start without timer=1 hung up my physical machine after daemon is started: Is there an oops in dmesg, or is the machine dead? What's the host kernel version? ยด Machine was dead. no infos inside dmesg / messages. We use FC9, Linux kvm03.dev.softwaredemo.de 2.6.27.21-78.2.41.fc9.x86_64 #1 SMP Mon Mar 23 23:22:15 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux regards Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
so, setup with opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start Use 'opreport -l'. Make sure your qemu isn't stripped. Build kvm-86 with ./configure ... --disable-strip and paused vm, now I got: opreport -l CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples %image name app name symbol name 112184 99.1919 no-vmlinux no-vmlinux (no symbols) 266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no symbols) 940.0831 libpthread-2.8.solibpthread-2.8.so __read_nocancel 720.0637 opreport opreport (no symbols) 630.0557 libc-2.8.so libc-2.8.so __select_nocancel 550.0486 libpthread-2.8.solibpthread-2.8.so __write_nocancel 370.0327 libpthread-2.8.solibpthread-2.8.so __pthread_enable_asynccancel 290.0256 libpthread-2.8.solibpthread-2.8.so __pthread_disable_asynccancel 280.0248 libpthread-2.8.solibpthread-2.8.so __libc_sigaction 280.0248 libpthread-2.8.solibpthread-2.8.so pthread_mutex_lock 210.0186 librt-2.8.so librt-2.8.so clock_gettime 200.0177 bash bash (no symbols) 190.0168 [vdso] (tgid:4783 range:0x7b3ff000-0x7b40) qemu-system-x86_64 (no symbols) 190.0168 libc-2.8.so libc-2.8.so __libc_disable_asynccancel 170.0150 librt-2.8.so librt-2.8.so timer_settime@@GLIBC_2.3.3 150.0133 libpthread-2.8.solibpthread-2.8.so __pthread_mutex_unlock_usercnt 150.0133 librt-2.8.so librt-2.8.so timer_gettime@@GLIBC_2.3.3 130.0115 oprofiledoprofiled(no symbols) 120.0106 libc-2.8.so libc-2.8.so __libc_enable_asynccancel 120.0106 libstdc++.so.6.0.10 libstdc++.so.6.0.10 (no symbols) 100.0088 libc-2.8.so libc-2.8.so _int_malloc 6 ... And top tell me PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4783 root 20 0 631m 536m 1956 S 7.0 6.7 0:22.94 qemu-system-x86 But the VM is paused - definitely .. Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Q: Stopped VM still using host cpu CPU ?
On 10/19/2009 03:47 AM, Daniel Schwager wrote: Build kvm-86 with ./configure ... --disable-strip and paused vm, now I got: opreport -l CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt samples %image name app name symbol name 112184 99.1919 no-vmlinux no-vmlinux (no symbols) 266 0.2352 qemu-system-x86_64 qemu-system-x86_64 (no symbols) Are you using the right qemu? Note 'no symbols'. Also, most samples come from the kernel, install the kernel-debuginfo package as well. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
On 10/04/2009 05:21 PM, Daniel Schwager wrote: How long is this after the 'stop'? 30 seconds or 2 days ... the process takes CPU all the time Can you take an oprofile run to see where it's spending its time? I'm not that familiar with oprofile ... so, setup with opcontrol --deinit; modprobe oprofile timer=1; opcontrol --start gave me the following result: kvm03:~# opreport CPU: CPU with timer interrupt, speed 0 MHz (estimated) Profiling through timer interrupt TIMER:0| samples| %| -- 613327 95.3150 no-vmlinux 12416 1.9295 qemu-system-x86_64 TIMER:0| samples| %| -- 10888 87.6933 qemu-system-x86_64 209 1.6833 [vdso] (tgid:2164 range:0x737ff000-0x7380) 171 1.3773 [vdso] (tgid:28126 range:0x7fffe7dff000-0x7fffe7e0) 168 1.3531 [vdso] (tgid:11172 range:0x767fe000-0x767ff000) 147 1.1840 [vdso] (tgid:1511 range:0x7fff741fe000-0x7fff741ff000) 143 1.1517 [vdso] (tgid:28327 range:0x7fffdcbfe000-0x7fffdcbff000) 141 1.1356 [vdso] (tgid:17010 range:0x7fff2a1fe000-0x7fff2a1ff000) 127 1.0229 [vdso] (tgid:27527 range:0x7fff2cffe000-0x7fff2cfff000) 111 0.8940 [vdso] (tgid:27644 range:0x7fffec7fe000-0x7fffec7ff000) 92 0.7410 [vdso] (tgid:11565 range:0x7fff247fe000-0x7fff247ff000) 81 0.6524 [vdso] (tgid:27438 range:0x7fff7c3fe000-0x7fff7c3ff000) 76 0.6121 [vdso] (tgid:26857 range:0x7a7fe000-0x7a7ff000) 62 0.4994 [vdso] (tgid:1676 range:0x7fff2f5ff000-0x7fff2f60) 11488 1.7853 libpthread-2.8.so Try to start without timer=1 hung up my physical machine after daemon is started: kvm03:/opt/kvm-86-vnc-patch# opcontrol --deinit; modprobe oprofile ; opcontrol --start Stopping profiling. Killing daemon. Unloading oprofile module Using default event: CPU_CLK_UNHALTED:10:0:1:1 Using 2.6+ OProfile kernel interface. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Could I give you more information ? Should I setup oprofile in another way ? regards Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Q: Stopped VM still using host cpu CPU ?
On 10/01/2009 12:32 PM, Daniel Schwager wrote: Hi, we are running some stopped (sending stop via kvm-monitor socket) vm's on our system. My intention was to pause (stop) the vm's and unpause (cont) them on demand (very fast, without time delay, within 2 seconds ..). After 'stop'ing, the vm's still using CPU-load, like the top will tell: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 25983 root 20 0 495m 407m 1876 R 8.9 2.5 228:09.15 qemu-system-x86 25523 root 20 0 495m 2040 1868 S 7.9 0.0 2700:16 qemu-system-x86 It shouldn't do that. How long is this after the 'stop'? Was the guest doing intensive I/O? Are you sure qemu responded to the stop command (by printing the monitor prompt)? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Q: Stopped VM still using host cpu CPU ?
On 10/01/2009 01:47 PM, Daniel Schwager wrote: If i send a signal STOP/CONT (kill -STOPpid or kill -CONTpid) to the KVM-process, it looks like the kvm does not (sure ;-) use any host CPU usage. - Are there some side effects using this approach ? (e.g. with networking, ...) The monitor, vnc, and sdl stop working. - And, why is there a way to send STOP/CONT via socket to KVM-process ? Why not using the sending-signal-apporch ? There's the 'stop' command... -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
After 'stop'ing, the vm's still using CPU-load, like the top will tell PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 25983 root 20 0 495m 407m 1876 R 8.9 2.5 228:09.15 qemu-system-x86 It shouldn't do that. ok. How long is this after the 'stop'? 30 seconds or 2 days ... the process takes CPU all the time Was the guest doing intensive I/O? No - normaly only a standby windows xp, without running programs Are you sure qemu responded to the stop command (by printing the monitor prompt)? sure. I use the socket communication to CONT the process - works well (-: Also, if a STOP the kvm, I can access the internal VNC-server (sure, mouse movements will NOT move the mouse - because it's stopped..) I'm using kvm-86. regard Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Q: Stopped VM still using host cpu CPU ?
On 10/04/2009 05:21 PM, Daniel Schwager wrote: How long is this after the 'stop'? 30 seconds or 2 days ... the process takes CPU all the time Can you take an oprofile run to see where it's spending its time? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Q: Stopped VM still using host cpu CPU ?
Hi, we are running some stopped (sending stop via kvm-monitor socket) vm's on our system. My intention was to pause (stop) the vm's and unpause (cont) them on demand (very fast, without time delay, within 2 seconds ..). After 'stop'ing, the vm's still using CPU-load, like the top will tell: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 25983 root 20 0 495m 407m 1876 R 8.9 2.5 228:09.15 qemu-system-x86 25523 root 20 0 495m 2040 1868 S 7.9 0.0 2700:16 qemu-system-x86 19738 root 20 0 1139m 40m 1876 R 7.3 0.3 2501:57 qemu-system-x86 32521 root 20 0 495m 407m 1876 S 6.9 2.5 136:23.23 qemu-system-x86 21773 root 20 0 495m 407m 1876 S 5.6 2.5 575:01.44 qemu-system-x86 6720 root 20 0 2168m 2.0g 1876 R 4.6 12.9 524:55.70 qemu-system-x86 6819 root 20 0 2168m 2.0g 1876 S 4.6 12.9 489:24.24 qemu-system-x86 12752 root 20 0 495m 407m 1876 S 4.6 2.5 0:44.57 qemu-system-x86 22803 root 20 0 1139m 211m 1868 S 4.0 1.3 1217:21 qemu-system-x86 27083 root 20 0 1139m 1.0g 1868 S 4.0 6.5 121:58.05 qemu-system-x86 21970 root 20 0 495m 407m 1876 S 3.6 2.5 434:48.36 qemu-system-x86 3649 root 20 0 496m 394m 1908 S 3.3 2.5 39:41.61 qemu-system-x86 19919 root 20 0 559m 402m 1880 S 3.3 2.5 469:57.14 qemu-system-x86 24461 root 20 0 495m 2048 1876 S 3.3 0.0 1081:41 qemu-system-x86 24619 root 20 0 688m 536m 1880 S 3.0 3.3 293:15.38 qemu-system-x86 24776 root 20 0 495m 407m 1856 S 3.0 2.5 92:08.95 qemu-system-x86 ... So, how can I prevent the paused/stopped VM's to use my CPU from the hostsystem ? Is there a way to handle this ? Sure, stop+migrating the vm to filesystem will solve my problem - but restart (migrate back from filesystem to RAM) takes a lot (30 seconds) of time. best regards Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Q: Stopped VM still using host cpu CPU ?
One more, So, how can I prevent the paused/stopped VM's to use my CPU from the hostsystem ? Is there a way to handle this ? If i send a signal STOP/CONT (kill -STOP pid or kill -CONT pid) to the KVM-process, it looks like the kvm does not (sure ;-) use any host CPU usage. - Are there some side effects using this approach ? (e.g. with networking, ...) - And, why is there a way to send STOP/CONT via socket to KVM-process ? Why not using the sending-signal-apporch ? best regards Danny -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
