Re: [kvm] Re: tcpdump locks up kvm host for a while.
On 10/05/2011 10:29 PM, Robin Lee Powell wrote: > > # > # (For a higher level overview, try: perf report --sort comm,dso) > # > > How helpful is that? -_- > > I'm guessing I need --guestkallsyms= ; since they're all the same > kernel I thought it'd figure it out. I'll redo. OK, here's a "better" version. # Events: 46K cycles # # Overhead CommandShared Object Symbol # ... ... # 74.81% qemu-kvm [unknown][u] 0x7fbdffd4c18a This is in userspace, so it seems the guest wasn't completely stuck. Try 'top -b' inside the guest to record what happens, let's see what processes this is and go from there. 25.14% qemu-kvm [guest.kernel.kallsyms] [g] 0x82f0 This doesn't resolve, please make sure the kernel-debuginfo package is installed in the guest and use the guestmount option. (or you can install it in the host, I think) -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Wed, Oct 05, 2011 at 11:36:53AM -0700, Robin Lee Powell wrote: > On Wed, Oct 05, 2011 at 08:21:53PM +0200, Avi Kivity wrote: > > On 10/04/2011 09:40 PM, Robin Lee Powell wrote: > > >> >When I run tcpdump on a *guest*, the entire guest completely > > >> >freezes up; no response even to hitting enter on the console. > > >> >"virsh list" also locks up whenever it tries to print state > > >> >about that VM (but the others work fine), as does any other > > >> >operation that touches the state of that VM. The VM takes up > > >> >100% of CPU on one core while this is happening. Eventually > > >> >it gets better. > > >> > > >> You can use 'perf kvm' to figure out where the guest is > > >> spinning. > > > > > >OK, gathered with: > > > > > >sudo perf kvm --guest --host record -o /tmp/kvm_perf -a > > > > > >I don't know how to read it at all, so it's at > > >http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf > > > > > > > Not accessible. > > -_- Fixed. > > > Please post the output of 'perf kvm report > log'. > > # Events: 42K > # > # Overhead Command Shared Object Symbol > # .. > # >100.00% qemu-kvm [unknown] [g] 0x8111c7a5 > > > # > # (For a higher level overview, try: perf report --sort comm,dso) > # > > How helpful is that? -_- > > I'm guessing I need --guestkallsyms= ; since they're all the same > kernel I thought it'd figure it out. I'll redo. OK, here's a "better" version. # Events: 46K cycles # # Overhead CommandShared Object Symbol # ... ... # 74.81% qemu-kvm [unknown][u] 0x7fbdffd4c18a 25.14% qemu-kvm [guest.kernel.kallsyms] [g] 0x82f0 0.03% qemu-kvm [virtio_net] [g] 0x83e8 0.01% qemu-kvm [virtio_balloon] [g] 0x103b 0.00% qemu-kvm [ip6_tables] [g] compat_standard_to_user 0.00% qemu-kvm [ipv6] [g] icmpv6_send 0.00% qemu-kvm [virtio_blk] [g] 0x7783 0.00% qemu-kvm [ipv6] [g] raw6_seq_show 0.00% qemu-kvm [ipv6] [g] icmpv6_rcv 0.00% qemu-kvm [virtio_net] [g] fini 0.00% qemu-kvm [ip6table_filter][g] 0x9b5 # # (For a higher level overview, try: perf report --sort comm,dso) # The file is also updated. -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tcpdump locks up kvm host for a while.
On 10/04/2011 09:40 PM, Robin Lee Powell wrote: > >When I run tcpdump on a *guest*, the entire guest completely > >freezes up; no response even to hitting enter on the console. > >"virsh list" also locks up whenever it tries to print state about > >that VM (but the others work fine), as does any other operation > >that touches the state of that VM. The VM takes up 100% of CPU > >on one core while this is happening. Eventually it gets better. > > You can use 'perf kvm' to figure out where the guest is spinning. OK, gathered with: sudo perf kvm --guest --host record -o /tmp/kvm_perf -a I don't know how to read it at all, so it's at http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf Not accessible. Please post the output of 'perf kvm report > log'. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tcpdump locks up kvm host for a while.
On Tue, Oct 04, 2011 at 08:15:12PM +0200, Avi Kivity wrote: > On 10/03/2011 05:03 AM, Robin Lee Powell wrote: > >> > > Alternatively, detailed instructions to reproduce? > >> > > >> > Start a VM. Wait a few days. Run tcpdump. The system > >> > locks up for 30+ seconds. > >> > >> How do you detect the lockup? Are you at the console when > >> this happens and everything just freezes? The entire host, > >> right, not guests? > >> > >> Is the lockup immediate after starting tcpdump? > > > >Ooooh. Oh, we're on the wrong page entirely. This *only* > >happens on the guests. tcpdump on the master host causes no > >trouble of any kind whatsoever. > > > >I guess I never specified that, maybe? Shame on me, if so. > > Well, the subject line does say "kvm host". *mega-face-palm* (the subject line is also filled with crap of my own devising; I think I've fixed the script in question) > >When I run tcpdump on a *guest*, the entire guest completely > >freezes up; no response even to hitting enter on the console. > >"virsh list" also locks up whenever it tries to print state about > >that VM (but the others work fine), as does any other operation > >that touches the state of that VM. The VM takes up 100% of CPU > >on one core while this is happening. Eventually it gets better. > > You can use 'perf kvm' to figure out where the guest is spinning. OK, gathered with: sudo perf kvm --guest --host record -o /tmp/kvm_perf -a I don't know how to read it at all, so it's at http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf > >> > That's all I have. There are, of course, any number of details that > >> > make my VMs different from yours, but they're basically straight > >> > Fedora 15. I could give arbitrary amounts of detail, but I've no > >> > idea what to concentrate on. > >> > > >> > >> host /proc/cpuinfo > > > >model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz > >flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > >cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm > >constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 > >monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow > > constant_tsc indicates tsc should be good. Yeah, that's what hte host is on. -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On 10/03/2011 05:03 AM, Robin Lee Powell wrote: > > > Alternatively, detailed instructions to reproduce? > > > > Start a VM. Wait a few days. Run tcpdump. The system locks up > > for 30+ seconds. > > How do you detect the lockup? Are you at the console when this > happens and everything just freezes? The entire host, right, not > guests? > > Is the lockup immediate after starting tcpdump? Ooooh. Oh, we're on the wrong page entirely. This *only* happens on the guests. tcpdump on the master host causes no trouble of any kind whatsoever. I guess I never specified that, maybe? Shame on me, if so. Well, the subject line does say "kvm host". If it was the *host*, I wouldn't be complaining on the KVM list, though, I'd be complaining on the main kernel list. :) Be sure to copy kvm@ too if it happens - certainly kvm bugs can affect the host. When I run tcpdump on a *guest*, the entire guest completely freezes up; no response even to hitting enter on the console. "virsh list" also locks up whenever it tries to print state about that VM (but the others work fine), as does any other operation that touches the state of that VM. The VM takes up 100% of CPU on one core while this is happening. Eventually it gets better. You can use 'perf kvm' to figure out where the guest is spinning. > > That's all I have. There are, of course, any number of details that > > make my VMs different from yours, but they're basically straight > > Fedora 15. I could give arbitrary amounts of detail, but I've no > > idea what to concentrate on. > > > > host /proc/cpuinfo model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow constant_tsc indicates tsc should be good. > qemu command line qemu 11811 1 1 Oct01 ?00:42:13 /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1536 -smp 1,sockets=1,cores=1,threads=1 -name vrici -uuid 20fc34fc-438e-3f51-d53e-406a68f96cbc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vrici.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/mapper/local-vrici_root,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/local-vrici_srv,if=none,id=drive-virtio-disk1,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/mapper/local-vrici_swap,if=none,id=drive-virtio-disk2,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/dev/mapper/local-vrici_home,if=none,id=drive-virtio-disk3,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -netdev tap,fd=32,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:34:91:67,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 Nothing suspicious here. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
> When I run tcpdump on a *guest*, the entire guest completely freezes > up; no response even to hitting enter on the console. "virsh list" > also locks up whenever it tries to print state about that VM (but > the others work fine), as does any other operation that touches the > state of that VM. The VM takes up 100% of CPU on one core while > this is happening. Eventually it gets better. > exactly the same I was describing here some time ago, with the difference I get a lot of ugly backtraces and sometimes the guest doesn't get to usable state at all (filesystems switches to read only due to errors or the whole guest locks up) n. pgpaWev9PeclA.pgp Description: PGP signature
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Sun, Oct 02, 2011 at 03:38:29PM -0400, Avi Kivity wrote: > > > > > > > Well, any messages? > > > > None I can find. I'll try again later today and double check. > > > Please do. Oct 2 19:53:01 vrici dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper) Oct 2 19:53:01 vrici dbus: [system] Successfully activated service 'net.reactivated.Fprint' Oct 2 19:53:55 vrici kernel: [155001.460802] device eth0 entered promiscuous mode Oct 2 19:53:56 vrici kernel: [155002.367313] device eth0 left promiscuous mode type=SYSCALL msg=audit(10/02/2011 19:53:14.731:48050) : arch=x86_64 syscall=execve success=yes exit=0 a0=7fff41959fd0 a1=7fc2f48d5150 a2=1474800 a3=6a6222 items=3 ppid=18453 pid=18456 auid=rlpowell uid=rlpowell gid=rlpowell euid=rlpowell suid=rlpowell fsuid=rlpowell egid=rlpowell sgid=rlpowell fsgid=rlpowell tty=pts0 ses=4 comm=date exe=/bin/date subj=staff_u:staff_r:staff_t:s0 key=64bit_execs type=SYSCALL msg=audit(10/02/2011 19:53:56.454:48059) : arch=x86_64 syscall=close success=yes exit=0 a0=3 a1=0 a2=1c11220 a3=7fff2eb675b0 items=0 ppid=18457 pid=18458 auid=rlpowell uid=tcpdump gid=tcpdump euid=tcpdump suid=tcpdump fsuid=tcpdump egid=tcpdump sgid=tcpdump fsgid=tcpdump tty=pts0 ses=4 comm=tcpdump exe=/usr/sbin/tcpdump subj=staff_u:unconfined_r:unconfined_t:s0 key=(null) type=ANOM_PROMISCUOUS msg=audit(10/02/2011 19:53:56.454:48059) : dev=eth0 prom=no old_prom=yes auid=rlpowell uid=tcpdump gid=tcpdump ses=4 That's all I could find that's even remotely interesting. > > > Alternatively, detailed instructions to reproduce? > > > > Start a VM. Wait a few days. Run tcpdump. The system locks up > > for 30+ seconds. > > How do you detect the lockup? Are you at the console when this > happens and everything just freezes? The entire host, right, not > guests? > > Is the lockup immediate after starting tcpdump? Ooooh. Oh, we're on the wrong page entirely. This *only* happens on the guests. tcpdump on the master host causes no trouble of any kind whatsoever. I guess I never specified that, maybe? Shame on me, if so. If it was the *host*, I wouldn't be complaining on the KVM list, though, I'd be complaining on the main kernel list. :) When I run tcpdump on a *guest*, the entire guest completely freezes up; no response even to hitting enter on the console. "virsh list" also locks up whenever it tries to print state about that VM (but the others work fine), as does any other operation that touches the state of that VM. The VM takes up 100% of CPU on one core while this is happening. Eventually it gets better. > > That's all I have. There are, of course, any number of details that > > make my VMs different from yours, but they're basically straight > > Fedora 15. I could give arbitrary amounts of detail, but I've no > > idea what to concentrate on. > > > > host /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz stepping: 6 cpu MHz : 2393.999 cache size : 4096 KB physical id : 0 siblings: 2 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow bogomips: 4787.99 clflush size: 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz stepping: 6 cpu MHz : 2393.999 cache size : 4096 KB physical id : 0 siblings: 2 core id : 1 cpu cores : 2 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow bogomips: 4787.78 clflush size: 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: > qemu command line qemu 11811 1 1 Oct01 ?00:42:13 /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1536 -smp 1,sockets=1,cores=1,threads=1 -name vrici -uuid 20fc34fc-438e-3f51-d53e-406a68f96cbc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vrici.monitor,server,nowait -mon chardev=charmon
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
> > > > Well, any messages? > > None I can find. I'll try again later today and double check. Please do. > > > Alternatively, detailed instructions to reproduce? > > Start a VM. Wait a few days. Run tcpdump. The system locks up for > 30+ seconds. How do you detect the lockup? Are you at the console when this happens and everything just freezes? The entire host, right, not guests? Is the lockup immediate after starting tcpdump? > > That's all I have. There are, of course, any number of details that > make my VMs different from yours, but they're basically straight > Fedora 15. I could give arbitrary amounts of detail, but I've no > idea what to concentrate on. > host /proc/cpuinfo qemu command line -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Sun, Oct 02, 2011 at 07:21:08PM +0200, Avi Kivity wrote: > On 10/02/2011 07:13 PM, Robin Lee Powell wrote: > >On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote: > >> On 10/02/2011 07:07 PM, Robin Lee Powell wrote: > >> >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote: > >> >> On 10/02/2011 02:45 AM, Robin Lee Powell wrote: > >> >> >I'm afraid clocksource=hpet didn't help much; after about 24 hours > >> >> >up, tcpdump hung the machine for for about 10 seconds. That may or > >> >> >may not be better than usual; I haven't been timing things > >> >> >precisely. My suspicion is that it's slightly better, but it's > >> >> >still not great. > >> >> > > >> >> >Also: > >> >> > > >> >> >rlpowell@vrici>dmesg | grep -i clock > >> >> >[0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM > >> rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us > >> console=ttyS0,115200 clocksource=hpet > >> >> >[0.00] kvm-clock: Using msrs 12 and 11 > >> >> >[0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock > >> >> >[0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock > >> >> >[0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS > >> rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 > >> KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >> >> >[0.00] hpet clockevent registered > >> >> >[0.260136] Switching to clocksource hpet > >> >> >[1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 > >> 07:49:50 UTC (1317455390) > >> >> >[1.821905] Refined TSC clocksource calibration: 2394.030 MHz. > >> >> >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) > >> >> > > >> >> >(note the last line) > >> >> > > >> >> > >> >> Do you get a similar error with kvmclock? > >> > > >> >I don't think I did, no. I'm not completely certain, but pretty > >> >sure. > >> > > >> > >> Then let's try to understand why it doesn't work with kvmclock. > > > >I'd love to. Suggestions? > > > > Well, any messages? None I can find. I'll try again later today and double check. > Alternatively, detailed instructions to reproduce? Start a VM. Wait a few days. Run tcpdump. The system locks up for 30+ seconds. That's all I have. There are, of course, any number of details that make my VMs different from yours, but they're basically straight Fedora 15. I could give arbitrary amounts of detail, but I've no idea what to concentrate on. -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On 10/02/2011 07:13 PM, Robin Lee Powell wrote: On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote: > On 10/02/2011 07:07 PM, Robin Lee Powell wrote: > >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote: > >> On 10/02/2011 02:45 AM, Robin Lee Powell wrote: > >> >I'm afraid clocksource=hpet didn't help much; after about 24 hours > >> >up, tcpdump hung the machine for for about 10 seconds. That may or > >> >may not be better than usual; I haven't been timing things > >> >precisely. My suspicion is that it's slightly better, but it's > >> >still not great. > >> > > >> >Also: > >> > > >> >rlpowell@vrici>dmesg | grep -i clock > >> >[0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >> >[0.00] kvm-clock: Using msrs 12 and 11 > >> >[0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock > >> >[0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock > >> >[0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >> >[0.00] hpet clockevent registered > >> >[0.260136] Switching to clocksource hpet > >> >[1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390) > >> >[1.821905] Refined TSC clocksource calibration: 2394.030 MHz. > >> >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) > >> > > >> >(note the last line) > >> > > >> > >> Do you get a similar error with kvmclock? > > > >I don't think I did, no. I'm not completely certain, but pretty > >sure. > > > > Then let's try to understand why it doesn't work with kvmclock. I'd love to. Suggestions? Well, any messages? Alternatively, detailed instructions to reproduce? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote: > On 10/02/2011 07:07 PM, Robin Lee Powell wrote: > >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote: > >> On 10/02/2011 02:45 AM, Robin Lee Powell wrote: > >> >I'm afraid clocksource=hpet didn't help much; after about 24 hours > >> >up, tcpdump hung the machine for for about 10 seconds. That may or > >> >may not be better than usual; I haven't been timing things > >> >precisely. My suspicion is that it's slightly better, but it's > >> >still not great. > >> > > >> >Also: > >> > > >> >rlpowell@vrici> dmesg | grep -i clock > >> >[0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM > >> rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us > >> console=ttyS0,115200 clocksource=hpet > >> >[0.00] kvm-clock: Using msrs 12 and 11 > >> >[0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock > >> >[0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock > >> >[0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS > >> rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 > >> KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >> >[0.00] hpet clockevent registered > >> >[0.260136] Switching to clocksource hpet > >> >[1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 > >> 07:49:50 UTC (1317455390) > >> >[1.821905] Refined TSC clocksource calibration: 2394.030 MHz. > >> >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) > >> > > >> >(note the last line) > >> > > >> > >> Do you get a similar error with kvmclock? > > > >I don't think I did, no. I'm not completely certain, but pretty > >sure. > > > > Then let's try to understand why it doesn't work with kvmclock. I'd love to. Suggestions? -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On 10/02/2011 07:07 PM, Robin Lee Powell wrote: On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote: > On 10/02/2011 02:45 AM, Robin Lee Powell wrote: > >I'm afraid clocksource=hpet didn't help much; after about 24 hours > >up, tcpdump hung the machine for for about 10 seconds. That may or > >may not be better than usual; I haven't been timing things > >precisely. My suspicion is that it's slightly better, but it's > >still not great. > > > >Also: > > > >rlpowell@vrici> dmesg | grep -i clock > >[0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >[0.00] kvm-clock: Using msrs 12 and 11 > >[0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock > >[0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock > >[0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet > >[0.00] hpet clockevent registered > >[0.260136] Switching to clocksource hpet > >[1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390) > >[1.821905] Refined TSC clocksource calibration: 2394.030 MHz. > >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) > > > >(note the last line) > > > > Do you get a similar error with kvmclock? I don't think I did, no. I'm not completely certain, but pretty sure. Then let's try to understand why it doesn't work with kvmclock. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote: > On 10/02/2011 02:45 AM, Robin Lee Powell wrote: > >I'm afraid clocksource=hpet didn't help much; after about 24 hours > >up, tcpdump hung the machine for for about 10 seconds. That may or > >may not be better than usual; I haven't been timing things > >precisely. My suspicion is that it's slightly better, but it's > >still not great. > > > >Also: > > > >rlpowell@vrici> dmesg | grep -i clock > >[0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD > >rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us > >console=ttyS0,115200 clocksource=hpet > >[0.00] kvm-clock: Using msrs 12 and 11 > >[0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock > >[0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock > >[0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM > >rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us > >console=ttyS0,115200 clocksource=hpet > >[0.00] hpet clockevent registered > >[0.260136] Switching to clocksource hpet > >[1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 > >UTC (1317455390) > >[1.821905] Refined TSC clocksource calibration: 2394.030 MHz. > >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) > > > >(note the last line) > > > > Do you get a similar error with kvmclock? I don't think I did, no. I'm not completely certain, but pretty sure. -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
On 10/02/2011 02:45 AM, Robin Lee Powell wrote: I'm afraid clocksource=hpet didn't help much; after about 24 hours up, tcpdump hung the machine for for about 10 seconds. That may or may not be better than usual; I haven't been timing things precisely. My suspicion is that it's slightly better, but it's still not great. Also: rlpowell@vrici> dmesg | grep -i clock [0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet [0.00] kvm-clock: Using msrs 12 and 11 [0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock [0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock [0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet [0.00] hpet clockevent registered [0.260136] Switching to clocksource hpet [1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390) [1.821905] Refined TSC clocksource calibration: 2394.030 MHz. [60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) (note the last line) Do you get a similar error with kvmclock? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.
I'm afraid clocksource=hpet didn't help much; after about 24 hours up, tcpdump hung the machine for for about 10 seconds. That may or may not be better than usual; I haven't been timing things precisely. My suspicion is that it's slightly better, but it's still not great. Also: rlpowell@vrici> dmesg | grep -i clock [0.00] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet [0.00] kvm-clock: Using msrs 12 and 11 [0.00] kvm-clock: cpu 0, msr 0:1b567c1, boot clock [0.00] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock [0.00] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet [0.00] hpet clockevent registered [0.260136] Switching to clocksource hpet [1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390) [1.821905] Refined TSC clocksource calibration: 2394.030 MHz. [60555.473797] Clocksource tsc unstable (delta = -42950328292 ns) (note the last line) -Robin On Sat, Oct 01, 2011 at 09:36:43AM +0200, Nikola Ciprich wrote: > well, give it a try ;) > I still haven't fully resolved this, but I'm sure it has to do something with > the timesource - with kvm-clock, it got much worse... > n. > > > [1.911056] Override clocksource tsc is not HRT compatible. Cannot > > switch while in HRT/NOHZ mode > > > > Will hpet do any better? > > > > -Robin > > > > -- > - > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28. rijna 168, 709 01 Ostrava > > tel.: +420 596 603 142 > fax:+420 596 621 273 > mobil: +420 777 093 799 > > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: ser...@linuxbox.cz > - -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [kvm] Re: tcpdump locks up kvm host for a while.
well, give it a try ;) I still haven't fully resolved this, but I'm sure it has to do something with the timesource - with kvm-clock, it got much worse... n. > [1.911056] Override clocksource tsc is not HRT compatible. Cannot switch > while in HRT/NOHZ mode > > Will hpet do any better? > > -Robin > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpFV1OCGVgYN.pgp Description: PGP signature
Re: [kvm] Re: tcpdump locks up kvm host for a while.
On Fri, Sep 30, 2011 at 09:03:21PM +0200, Nikola Ciprich wrote: > Hello Robin, > > actually I'm stil hitting problems with tcpdump in KVM virtuals.. > what is the content of > /sys/devices/system/clocksource/clocksource0/current_clocksource? > I guess it'll be kvm-clock, try using clocksource=tsc or hpet kernel > parameter. > does it help? [1.911056] Override clocksource tsc is not HRT compatible. Cannot switch while in HRT/NOHZ mode Will hpet do any better? -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: tcpdump locks up kvm host for a while.
Hello Robin, actually I'm stil hitting problems with tcpdump in KVM virtuals.. what is the content of /sys/devices/system/clocksource/clocksource0/current_clocksource? I guess it'll be kvm-clock, try using clocksource=tsc or hpet kernel parameter. does it help? n. > > -Robin > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- - Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - pgpHctssDgghH.pgp Description: PGP signature
Re: tcpdump locks up kvm host for a while.
On Tue, Sep 27, 2011 at 07:01:37PM -0700, Robin Lee Powell wrote: > > I can essentially test this at will, so feel free to ask for > debugging steps. > > My host and VMs are all Fedora 15. > > Every time I run tcpdump on a VM, it hangs for a while. It uses > all available CPU, "virsh list" hangs trying to show it, and I > can't shut it down except by killing the qemu-kvm process. > > After a few minutes, which amount of time seems to be > proportionate to how long it's been since I last ran tcpdump (the > longer it's been, the longer I have to wait) everything goes back > to normal if I don't kill the VM. > > Any idea what's going on there? Nobody else has this problem? How odd. -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
tcpdump locks up kvm host for a while.
I can essentially test this at will, so feel free to ask for debugging steps. My host and VMs are all Fedora 15. Every time I run tcpdump on a VM, it hangs for a while. It uses all available CPU, "virsh list" hangs trying to show it, and I can't shut it down except by killing the qemu-kvm process. After a few minutes, which amount of time seems to be proportionate to how long it's been since I last ran tcpdump (the longer it's been, the longer I have to wait) everything goes back to normal if I don't kill the VM. Any idea what's going on there? -Robin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
