On Wed, Sep 14, 2011 at 5:02 PM, William Cohen <wco...@redhat.com> wrote: > On 09/14/2011 10:51 AM, Stefan Hajnoczi wrote: >> On Tue, Sep 13, 2011 at 8:38 PM, William Cohen <wco...@redhat.com> wrote: >>> On 09/13/2011 12:10 PM, William Cohen wrote: >>> >>>> Should the qemu.kvm.cpu_in and qemu.kvm.cpu_out match up? There are a lot >>>> more qemu.kvm.cpu_out than qemu.kvm.cpu_in count. >>> >>> I found that cpu_in and cpu_out refer to input and output instructions. I >>> wrote a little script tally up the input and output operations on each port >>> to run on a qemu on fc15 machine. >>> >>> It generates output like the following: >>> >>> >>> cpu_in >>> port count >>> 0x01f7 3000 >>> 0x03d5 120 >>> 0xc000 2000 >>> 0xc002 3000 >>> >>> cpu_out >>> port count >>> 0x0080 480 >>> 0x01f1 2000 >>> 0x01f2 2000 >>> 0x01f3 2000 >>> 0x01f4 2000 >>> 0x01f5 2000 >>> 0x01f6 2000 >>> 0x01f7 1000 >>> 0x03d4 480 >>> 0x03d5 120 >>> 0x03f6 1000 >>> 0xc000 3000 >>> 0xc002 2000 >>> 0xc004 1000 >>> 0xc090 4 >>> >>> Looks like lots of touching the ide device ports (0x01f0-0x01ff) and some >>> vga controller (0x03d0-0x3df). This is kind of what would be expected when >>> the machine is doing a fsck and selinux relabel on the guest virtual >>> machines. Look like some pci device access >>> (http://www.tech-pro.net/intro_pci.html) also. >> >> I think the cpu_in/cpu_out tracing script can be useful in identifying >> performance problems, especially when obscure guest OSes experience >> poor performance due to weird ways of interacting with hardware. > > Glad this example looks useful. > >> Where are you putting your SystemTap scripts? I suggest creating a >> public git repo and adding a link from the QEMU wiki tracing page: >> http://wiki.qemu.org/Features/Tracing > > Definitely need a more lasting place to put the QEMU examples. SystemTap has > an examples directory which try to put things like this into > (http://sourceware.org/systemtap/examples/). These are run as part of the > testsuite. These qemu examples are not ready for inclusion in the examples. I > have set up https://github.com/wcohen/qemu_systemtap for the qemu examples.
Added a link from http://qemu.org/Features/Tracing. >> Perhaps we could even include them in a contrib/ or similar directory >> in qemu.git so that distros can ship them. But before we worry about >> that we need a useful set of things that can be observed. > > We definitely want to develop scripts that give people a better understanding > what is going on. I am just now learning how kvm and qemu work, so the early > scripts are just counting events to give people an idea events are frequent. > In some cases these could point to issues where some assumptions about event > frequency are incorrect. I would like to have scripts that examine the > sequences of events and indicate when long latencies occur, but I don't yet > have enough understanding of the how kvm and qemu work to implement those. The vmexit/entry things I mentioned in the mail above are events that can be traced in the host kernel (kvm:kvm_exit and kvm:kvm_entry). The QEMU-side equivalent is the ioctl(KVM_RUN) but we do not have any trace events in kvm-all.c yet (!!). A trace event before the ioctl and one after could be used to trace vcpu code execution from QEMU's perspective. Stefan
