[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/22/07, Dor Laor <[EMAIL PROTECTED]> wrote: > >> > >>> This is QEMU, with dynticks and HPET: > >> > >>> > >> > >>> % time seconds usecs/call callserrors syscall > >> > >>> -- --- --- - - > - > >--- > >> > >>> 52.100.002966 0 96840 > clock_gettime > >> > >>> 19.500.001110 0 37050 > timer_gettime > >> > >>> 10.660.000607 0 20086 > timer_settime > >> > >>> 10.400.000592 0 8985 2539 sigreturn > >> > >>> 4.940.000281 0 8361 2485 select > >> > >>> 2.410.000137 0 8362 gettimeofday > >> > >>> -- --- --- - - > - > >--- > >> > >>> 100.000.005693179684 5024 total > >> > >>> > >> > >>> > >> > >> This looks like 250 Hz? > >> > >> > >> > > > >> > > Nope: > >> > > > >> > > # CONFIG_NO_HZ is not set > >> > > # CONFIG_HZ_100 is not set > >> > > # CONFIG_HZ_250 is not set > >> > > # CONFIG_HZ_300 is not set > >> > > CONFIG_HZ_1000=y > >> > > CONFIG_HZ=1000 > >> > > > >> > > and I'm reading it from /proc/config.gz on the guest. > >> > > > >> > > >> > Yeah, thought so -- so dyntick is broken at present. > >> > >> I see a lot of sub ms timer_settime(). Many of them are the result of > >> ->expire_time being less than the current qemu_get_clock(). This > >> results into 250us timer due to MIN_TIMER_REARM_US; this happens only > >> for the REALTIME timer. Other sub-ms timers are generated by the > >> VIRTUAL timer. > >> > >> This first issue is easily fixed; if expire_time < current time then > >> the timer has expired and hasn't been reprogrammed (and thus can be > >> ignored). > >> VIRTUAL just becomes more accurate with dyntics, before multiple > >> timers were batched together. > >> > >> > Or maybe your host kernel can't support such a high rate. > >> > >> I don't know... a simple printf tells me that the signal handler is > >> called about 1050 times per second, which sounds about right. > > > >...unless strace is attached. ptrace()'ing the process really screw up > >the timing with dynticks; HPET is also affected but the performance > >hit is not as severe. > > > I didn't figure out how you use both hpet and dyn-tick together. I don't. Only one timer source is active at any time; the selection is done at startup with -clock option. > Hpet has periodic timer while dyn-tick is one shot timer each time. > Is ther a chance that both are working and that's the source of our > problems? No, the various sources are exclusive (though it might be possible to use HPET in one shot mode). Luca
[Qemu-devel] RE: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
>> > >>> This is QEMU, with dynticks and HPET: >> > >>> >> > >>> % time seconds usecs/call callserrors syscall >> > >>> -- --- --- - - - >--- >> > >>> 52.100.002966 0 96840 clock_gettime >> > >>> 19.500.001110 0 37050 timer_gettime >> > >>> 10.660.000607 0 20086 timer_settime >> > >>> 10.400.000592 0 8985 2539 sigreturn >> > >>> 4.940.000281 0 8361 2485 select >> > >>> 2.410.000137 0 8362 gettimeofday >> > >>> -- --- --- - - - >--- >> > >>> 100.000.005693179684 5024 total >> > >>> >> > >>> >> > >> This looks like 250 Hz? >> > >> >> > > >> > > Nope: >> > > >> > > # CONFIG_NO_HZ is not set >> > > # CONFIG_HZ_100 is not set >> > > # CONFIG_HZ_250 is not set >> > > # CONFIG_HZ_300 is not set >> > > CONFIG_HZ_1000=y >> > > CONFIG_HZ=1000 >> > > >> > > and I'm reading it from /proc/config.gz on the guest. >> > > >> > >> > Yeah, thought so -- so dyntick is broken at present. >> >> I see a lot of sub ms timer_settime(). Many of them are the result of >> ->expire_time being less than the current qemu_get_clock(). This >> results into 250us timer due to MIN_TIMER_REARM_US; this happens only >> for the REALTIME timer. Other sub-ms timers are generated by the >> VIRTUAL timer. >> >> This first issue is easily fixed; if expire_time < current time then >> the timer has expired and hasn't been reprogrammed (and thus can be >> ignored). >> VIRTUAL just becomes more accurate with dyntics, before multiple >> timers were batched together. >> >> > Or maybe your host kernel can't support such a high rate. >> >> I don't know... a simple printf tells me that the signal handler is >> called about 1050 times per second, which sounds about right. > >...unless strace is attached. ptrace()'ing the process really screw up >the timing with dynticks; HPET is also affected but the performance >hit is not as severe. > >Luca I didn't figure out how you use both hpet and dyn-tick together. Hpet has periodic timer while dyn-tick is one shot timer each time. Is ther a chance that both are working and that's the source of our problems? Dor
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On Wed, Aug 22, 2007 at 06:38:18PM +0200, Luca wrote: > and I'm reading it from /proc/config.gz on the guest. > > > And a huge number of settime calls? > > Yes, maybe some QEMU timer is using an interval < 1ms? > Dan do you any any idea of what's going on? Not really...
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/22/07, Luca <[EMAIL PROTECTED]> wrote: > On 8/22/07, Avi Kivity <[EMAIL PROTECTED]> wrote: > > Luca wrote: > > >>> This is QEMU, with dynticks and HPET: > > >>> > > >>> % time seconds usecs/call callserrors syscall > > >>> -- --- --- - - > > >>> 52.100.002966 0 96840 clock_gettime > > >>> 19.500.001110 0 37050 timer_gettime > > >>> 10.660.000607 0 20086 timer_settime > > >>> 10.400.000592 0 8985 2539 sigreturn > > >>> 4.940.000281 0 8361 2485 select > > >>> 2.410.000137 0 8362 gettimeofday > > >>> -- --- --- - - > > >>> 100.000.005693179684 5024 total > > >>> > > >>> > > >> This looks like 250 Hz? > > >> > > > > > > Nope: > > > > > > # CONFIG_NO_HZ is not set > > > # CONFIG_HZ_100 is not set > > > # CONFIG_HZ_250 is not set > > > # CONFIG_HZ_300 is not set > > > CONFIG_HZ_1000=y > > > CONFIG_HZ=1000 > > > > > > and I'm reading it from /proc/config.gz on the guest. > > > > > > > Yeah, thought so -- so dyntick is broken at present. > > I see a lot of sub ms timer_settime(). Many of them are the result of > ->expire_time being less than the current qemu_get_clock(). This > results into 250us timer due to MIN_TIMER_REARM_US; this happens only > for the REALTIME timer. Other sub-ms timers are generated by the > VIRTUAL timer. > > This first issue is easily fixed; if expire_time < current time then > the timer has expired and hasn't been reprogrammed (and thus can be > ignored). > VIRTUAL just becomes more accurate with dyntics, before multiple > timers were batched together. > > > Or maybe your host kernel can't support such a high rate. > > I don't know... a simple printf tells me that the signal handler is > called about 1050 times per second, which sounds about right. ...unless strace is attached. ptrace()'ing the process really screw up the timing with dynticks; HPET is also affected but the performance hit is not as severe. Luca
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/22/07, Luca <[EMAIL PROTECTED]> wrote: > I see a lot of sub ms timer_settime(). Many of them are the result of > ->expire_time being less than the current qemu_get_clock(). False alarm, this was a bug in the debug code :D Luca
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/22/07, Avi Kivity <[EMAIL PROTECTED]> wrote: > Luca wrote: > >>> This is QEMU, with dynticks and HPET: > >>> > >>> % time seconds usecs/call callserrors syscall > >>> -- --- --- - - > >>> 52.100.002966 0 96840 clock_gettime > >>> 19.500.001110 0 37050 timer_gettime > >>> 10.660.000607 0 20086 timer_settime > >>> 10.400.000592 0 8985 2539 sigreturn > >>> 4.940.000281 0 8361 2485 select > >>> 2.410.000137 0 8362 gettimeofday > >>> -- --- --- - - > >>> 100.000.005693179684 5024 total > >>> > >>> > >> This looks like 250 Hz? > >> > > > > Nope: > > > > # CONFIG_NO_HZ is not set > > # CONFIG_HZ_100 is not set > > # CONFIG_HZ_250 is not set > > # CONFIG_HZ_300 is not set > > CONFIG_HZ_1000=y > > CONFIG_HZ=1000 > > > > and I'm reading it from /proc/config.gz on the guest. > > > > Yeah, thought so -- so dyntick is broken at present. I see a lot of sub ms timer_settime(). Many of them are the result of ->expire_time being less than the current qemu_get_clock(). This results into 250us timer due to MIN_TIMER_REARM_US; this happens only for the REALTIME timer. Other sub-ms timers are generated by the VIRTUAL timer. This first issue is easily fixed; if expire_time < current time then the timer has expired and hasn't been reprogrammed (and thus can be ignored). VIRTUAL just becomes more accurate with dyntics, before multiple timers were batched together. > Or maybe your host kernel can't support such a high rate. I don't know... a simple printf tells me that the signal handler is called about 1050 times per second, which sounds about right. Luca
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Luca wrote: >>> This is QEMU, with dynticks and HPET: >>> >>> % time seconds usecs/call callserrors syscall >>> -- --- --- - - >>> 52.100.002966 0 96840 clock_gettime >>> 19.500.001110 0 37050 timer_gettime >>> 10.660.000607 0 20086 timer_settime >>> 10.400.000592 0 8985 2539 sigreturn >>> 4.940.000281 0 8361 2485 select >>> 2.410.000137 0 8362 gettimeofday >>> -- --- --- - - >>> 100.000.005693179684 5024 total >>> >>> >> This looks like 250 Hz? >> > > Nope: > > # CONFIG_NO_HZ is not set > # CONFIG_HZ_100 is not set > # CONFIG_HZ_250 is not set > # CONFIG_HZ_300 is not set > CONFIG_HZ_1000=y > CONFIG_HZ=1000 > > and I'm reading it from /proc/config.gz on the guest. > Yeah, thought so -- so dyntick is broken at present. Or maybe your host kernel can't support such a high rate. Probably needs hrtimers or qemu dyntick over hpet oneshot support. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/22/07, Avi Kivity <[EMAIL PROTECTED]> wrote: > Luca Tettamanti wrote: > > Il Wed, Aug 22, 2007 at 08:02:07AM +0300, Avi Kivity ha scritto: > > > >> Luca Tettamanti wrote: > >> > >> > >>> Actually I'm having troubles with cyclesoak (probably it's calibration), > >>> numbers are not very stable across multiple runs... > >>> > >>> > >> I've had good results with cyclesoak; maybe you need to run it in > >> runlevel 3 so the load generated by moving the mouse or breathing > >> doesn't affect meaurements. > >> > > > > This is what I did, I tested with -no-grapich in text console. > > Okay. Maybe cpu frequency scaling confused it then. Or something else? I set it performance, frequency was locked at 2.1GHz. > >>> The guest is an idle kernel with HZ=1000. > >>> > >>> > >> Can you double check this? The dyntick results show that this is either > >> a 100Hz kernel, or that there is a serious bug in dynticks. > >> > > > > Ops I sent the wrong files, sorry. > > > > This is QEMU, with dynticks and HPET: > > > > % time seconds usecs/call callserrors syscall > > -- --- --- - - > > 52.100.002966 0 96840 clock_gettime > > 19.500.001110 0 37050 timer_gettime > > 10.660.000607 0 20086 timer_settime > > 10.400.000592 0 8985 2539 sigreturn > > 4.940.000281 0 8361 2485 select > > 2.410.000137 0 8362 gettimeofday > > -- --- --- - - > > 100.000.005693179684 5024 total > > > > This looks like 250 Hz? Nope: # CONFIG_NO_HZ is not set # CONFIG_HZ_100 is not set # CONFIG_HZ_250 is not set # CONFIG_HZ_300 is not set CONFIG_HZ_1000=y CONFIG_HZ=1000 and I'm reading it from /proc/config.gz on the guest. > And a huge number of settime calls? Yes, maybe some QEMU timer is using an interval < 1ms? Dan do you any any idea of what's going on? Luca
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Luca Tettamanti wrote: > Il Wed, Aug 22, 2007 at 08:02:07AM +0300, Avi Kivity ha scritto: > >> Luca Tettamanti wrote: >> >> >>> Actually I'm having troubles with cyclesoak (probably it's calibration), >>> numbers are not very stable across multiple runs... >>> >>> >> I've had good results with cyclesoak; maybe you need to run it in >> runlevel 3 so the load generated by moving the mouse or breathing >> doesn't affect meaurements. >> > > This is what I did, I tested with -no-grapich in text console. > > Okay. Maybe cpu frequency scaling confused it then. Or something else? >>> The guest is an idle kernel with HZ=1000. >>> >>> >> Can you double check this? The dyntick results show that this is either >> a 100Hz kernel, or that there is a serious bug in dynticks. >> > > Ops I sent the wrong files, sorry. > > This is QEMU, with dynticks and HPET: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 52.100.002966 0 96840 clock_gettime > 19.500.001110 0 37050 timer_gettime > 10.660.000607 0 20086 timer_settime > 10.400.000592 0 8985 2539 sigreturn > 4.940.000281 0 8361 2485 select > 2.410.000137 0 8362 gettimeofday > -- --- --- - - > 100.000.005693179684 5024 total > This looks like 250 Hz? And a huge number of settime calls? Something's broken with dynticks. > % time seconds usecs/call callserrors syscall > -- --- --- - - > 93.370.025541 3 10194 10193 select > 4.820.001319 0 33259 clock_gettime > 1.100.000301 0 10195 gettimeofday > 0.710.000195 0 10196 10194 sigreturn > -- --- --- - - > 100.000.027356 63844 20387 total > This is expected and sane. > And this KVM: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 42.660.002885 0 4552724 ioctl > 25.620.001733 0 89305 clock_gettime > 13.120.000887 0 34894 timer_gettime > 7.970.000539 0 18016 timer_settime > 4.700.000318 0 12224 7270 rt_sigtimedwait > 2.790.000189 0 7271 select > 1.860.000126 0 7271 gettimeofday > 1.270.86 0 4954 rt_sigaction > -- --- --- - - > 100.000.006763219462 7294 total > Similarly broken. The effective frequency is twice qemu's. I think we had the same effect last time. > % time seconds usecs/call callserrors syscall > -- --- --- - - > 49.410.004606 0 5990027 ioctl > 24.140.002250 0 31252 21082 rt_sigtimedwait > 9.650.000900 0 51856 clock_gettime > 8.440.000787 0 17819 select > 4.420.000412 0 17819 gettimeofday > 3.940.000367 0 10170 rt_sigaction > -- --- --- - - > 100.000.009322188816 21109 total > > Similarly sane. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Il Wed, Aug 22, 2007 at 08:02:07AM +0300, Avi Kivity ha scritto: > Luca Tettamanti wrote: > > > Actually I'm having troubles with cyclesoak (probably it's calibration), > > numbers are not very stable across multiple runs... > > > > I've had good results with cyclesoak; maybe you need to run it in > runlevel 3 so the load generated by moving the mouse or breathing > doesn't affect meaurements. This is what I did, I tested with -no-grapich in text console. > > The guest is an idle kernel with HZ=1000. > > > > Can you double check this? The dyntick results show that this is either > a 100Hz kernel, or that there is a serious bug in dynticks. Ops I sent the wrong files, sorry. This is QEMU, with dynticks and HPET: % time seconds usecs/call callserrors syscall -- --- --- - - 52.100.002966 0 96840 clock_gettime 19.500.001110 0 37050 timer_gettime 10.660.000607 0 20086 timer_settime 10.400.000592 0 8985 2539 sigreturn 4.940.000281 0 8361 2485 select 2.410.000137 0 8362 gettimeofday -- --- --- - - 100.000.005693179684 5024 total % time seconds usecs/call callserrors syscall -- --- --- - - 93.370.025541 3 10194 10193 select 4.820.001319 0 33259 clock_gettime 1.100.000301 0 10195 gettimeofday 0.710.000195 0 10196 10194 sigreturn -- --- --- - - 100.000.027356 63844 20387 total And this KVM: % time seconds usecs/call callserrors syscall -- --- --- - - 42.660.002885 0 4552724 ioctl 25.620.001733 0 89305 clock_gettime 13.120.000887 0 34894 timer_gettime 7.970.000539 0 18016 timer_settime 4.700.000318 0 12224 7270 rt_sigtimedwait 2.790.000189 0 7271 select 1.860.000126 0 7271 gettimeofday 1.270.86 0 4954 rt_sigaction -- --- --- - - 100.000.006763219462 7294 total % time seconds usecs/call callserrors syscall -- --- --- - - 49.410.004606 0 5990027 ioctl 24.140.002250 0 31252 21082 rt_sigtimedwait 9.650.000900 0 51856 clock_gettime 8.440.000787 0 17819 select 4.420.000412 0 17819 gettimeofday 3.940.000367 0 10170 rt_sigaction -- --- --- - - 100.000.009322188816 21109 total Luca -- Runtime error 6D at f000:a12f : user incompetente
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Luca Tettamanti wrote: > Actually I'm having troubles with cyclesoak (probably it's calibration), > numbers are not very stable across multiple runs... > I've had good results with cyclesoak; maybe you need to run it in runlevel 3 so the load generated by moving the mouse or breathing doesn't affect meaurements. > I've also tried APC which was suggested by malc[1] and: > - readings are far more stable > - the gap between dynticks and non-dynticks seems not significant > > >> Can you verify this by running >> >>strace -c -p `pgrep qemu` & sleep 10; pkill strace >> >> for all 4 cases, and posting the results? >> > > Plain QEMU: > > With dynticks: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 57.970.000469 0 13795 clock_gettime > 32.880.000266 0 1350 gettimeofday > 7.420.60 0 1423 1072 sigreturn > 1.730.14 0 5049 timer_gettime > 0.000.00 0 1683 1072 select > 0.000.00 0 2978 timer_settime > -- --- --- - - > 100.000.000809 26278 2144 total > The 1072 select() errors are the delivered ticks (EINTR). But why only 1000? would have expected 1 for a 1000Hz guest in a 10 sec period. > HPET: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 87.480.010459 1 10381 10050 select > 8.450.001010 0 40736 clock_gettime > 2.730.000326 0 10049 gettimeofday > 1.350.000161 0 10086 10064 sigreturn > -- --- --- - - > 100.000.011956 71252 20114 total > This is expected. 1 tick per millisecond. > Unix (SIGALRM): > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 90.360.011663 1 10291 9959 select > 7.380.000953 0 40355 clock_gettime > 2.050.000264 0 9960 gettimeofday > 0.210.27 0 9985 9969 sigreturn > -- --- --- - - > 100.000.012907 70591 19928 total > Same here. > And KVM: > > dynticks: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 78.900.004001 1 6681 5088 rt_sigtimedwait > 10.870.000551 0 27901 clock_gettime > 4.930.000250 0 7622 timer_settime > 4.300.000218 0 10078 timer_gettime > 0.390.20 0 3863 gettimeofday > 0.350.18 0 6054 ioctl > 0.260.13 0 4196 select > 0.000.00 0 1593 rt_sigaction > -- --- --- - - > 100.000.005071 67988 5088 total > kvm uses sigtimedwait() to wait for signals. Here, an error (ETIMEDOUT) indicates we did _not_ get a wakeup, so there are 1500 wakeups in a 10 second period. Strange. Some calibration error? > HPET: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 90.200.011029 0 32437 22244 rt_sigtimedwait > 4.460.000545 0 44164 clock_gettime > 2.590.000317 0 12128 gettimeofday > 1.500.000184 0 10193 rt_sigaction > 1.100.000134 0 12461 select > 0.150.18 0 6060 ioctl > -- --- --- - - > 100.000.012227117443 22244 total > 10K wakeups per second. The code is not particularly efficient (11 syscalls per tick), but overhead is still low. > Unix: > > % time seconds usecs/call callserrors syscall > -- --- --- - - > 83.290.012522 0 31652 21709 rt_sigtimedwait > 6.910.001039 0 43125 clock_gettime > 3.500.000526 0 6042 ioctl > 2.740.000412 0 9943 rt_sigaction > 1.980.000298 0 12183 select > 1.580.000238 0 11850 gettimeofday > -- --- --- - - > 100.00
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On Tue, 21 Aug 2007, Luca Tettamanti wrote: Avi Kivity ha scritto: Luca Tettamanti wrote: At 1000Hz: QEMU hpet5.5% dynticks 11.7% KVM hpet3.4% dynticks7.3% No surprises here, you can see the additional 1k syscalls per second. This is very surprising to me. The 6.2% difference for the qemu case translates to 62ms per second, or 62us per tick at 1000Hz. That's more than a hundred simple syscalls on modern processors. We shouldn't have to issue a hundred syscalls per guest clock tick. [..snip preulde..] I've also tried APC which was suggested by malc[1] and: - readings are far more stable - the gap between dynticks and non-dynticks seems not significant [..dont snip the obvious fact and snip the numbers..] Luca [1] copy_to_user inside spinlock is a big no-no ;) [..notice a projectile targeting at you and rush to see the code..] Mixed feelings about this... But in principle the code ofcourse is dangerous, thank you kindly for pointing this out. I see two ways out of this: a. moving the lock/unlock inside the loop with unlock preceding sometimes sleep deprived copy_to_user b. fill temporaries and after the loop is done copy it in one go Too late, too hot, i wouldn't mind beying on a receiving side of a good advice. -- vale
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Avi Kivity ha scritto: > Luca Tettamanti wrote: >> At 1000Hz: >> >> QEMU >> hpet5.5% >> dynticks 11.7% >> >> KVM >> hpet3.4% >> dynticks7.3% >> >> No surprises here, you can see the additional 1k syscalls per second. > > This is very surprising to me. The 6.2% difference for the qemu case > translates to 62ms per second, or 62us per tick at 1000Hz. That's more > than a hundred simple syscalls on modern processors. We shouldn't have to > issue a hundred syscalls per guest clock tick. APIC or PIT interrupts are delivered using the timer, which will be re-armed after each tick, so I'd expect 1k timer_settime per second. But according to strace it's not happening, maybe I'm misreading the code? > The difference with kvm is smaller (just 3.9%), which is not easily > explained as the time for the extra syscalls should be about the same. My > guess is that guest behavior is different; with dynticks the guest does > about twice as much work as with hpet. Actually I'm having troubles with cyclesoak (probably it's calibration), numbers are not very stable across multiple runs... I've also tried APC which was suggested by malc[1] and: - readings are far more stable - the gap between dynticks and non-dynticks seems not significant > Can you verify this by running > >strace -c -p `pgrep qemu` & sleep 10; pkill strace > > for all 4 cases, and posting the results? Plain QEMU: With dynticks: % time seconds usecs/call callserrors syscall -- --- --- - - 57.970.000469 0 13795 clock_gettime 32.880.000266 0 1350 gettimeofday 7.420.60 0 1423 1072 sigreturn 1.730.14 0 5049 timer_gettime 0.000.00 0 1683 1072 select 0.000.00 0 2978 timer_settime -- --- --- - - 100.000.000809 26278 2144 total HPET: % time seconds usecs/call callserrors syscall -- --- --- - - 87.480.010459 1 10381 10050 select 8.450.001010 0 40736 clock_gettime 2.730.000326 0 10049 gettimeofday 1.350.000161 0 10086 10064 sigreturn -- --- --- - - 100.000.011956 71252 20114 total Unix (SIGALRM): % time seconds usecs/call callserrors syscall -- --- --- - - 90.360.011663 1 10291 9959 select 7.380.000953 0 40355 clock_gettime 2.050.000264 0 9960 gettimeofday 0.210.27 0 9985 9969 sigreturn -- --- --- - - 100.000.012907 70591 19928 total And KVM: dynticks: % time seconds usecs/call callserrors syscall -- --- --- - - 78.900.004001 1 6681 5088 rt_sigtimedwait 10.870.000551 0 27901 clock_gettime 4.930.000250 0 7622 timer_settime 4.300.000218 0 10078 timer_gettime 0.390.20 0 3863 gettimeofday 0.350.18 0 6054 ioctl 0.260.13 0 4196 select 0.000.00 0 1593 rt_sigaction -- --- --- - - 100.000.005071 67988 5088 total HPET: % time seconds usecs/call callserrors syscall -- --- --- - - 90.200.011029 0 32437 22244 rt_sigtimedwait 4.460.000545 0 44164 clock_gettime 2.590.000317 0 12128 gettimeofday 1.500.000184 0 10193 rt_sigaction 1.100.000134 0 12461 select 0.150.18 0 6060 ioctl -- --- --- - - 100.000.012227117443 22244 total Unix: % time seconds usecs/call callserrors syscall -- --- --- - - 83.290.012522 0 31652 21709 rt_sigtimedwait 6.910.001039 0 43125 clock_gettime 3.500.000526 0 6042 ioctl 2.740.000412 0 9943 rt_sigaction 1.980.000298 0 12183 select 1.580.000238
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Luca Tettamanti wrote: Run a 100Hz guest, measure cpu usage using something accurate like cyclesoak, with and without dynticks, with and without kvm. Ok, here I've measured the CPU usage on the host when running an idle guest. At 100Hz QEMU hpet4.8% dynticks5.1% Note: I've taken the mean over a period of 20 secs, but the difference between hpet and dynticks is well inside the variability of the test. KVM hpet2.2% dynticks1.0% Hum... here the numbers jumps a bit, but dynticks is always below hpet. The differences here are small, so I'll focus on the 1000Hz case. At 1000Hz: QEMU hpet5.5% dynticks 11.7% KVM hpet3.4% dynticks7.3% No surprises here, you can see the additional 1k syscalls per second. This is very surprising to me. The 6.2% difference for the qemu case translates to 62ms per second, or 62us per tick at 1000Hz. That's more than a hundred simple syscalls on modern processors. We shouldn't have to issue a hundred syscalls per guest clock tick. The difference with kvm is smaller (just 3.9%), which is not easily explained as the time for the extra syscalls should be about the same. My guess is that guest behavior is different; with dynticks the guest does about twice as much work as with hpet. Can you verify this by running strace -c -p `pgrep qemu` & sleep 10; pkill strace for all 4 cases, and posting the results? -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On Mon, 20 Aug 2007, Luca Tettamanti wrote:
Il Sun, Aug 19, 2007 at 10:31:26PM +0300, Avi Kivity ha scritto:
Luca wrote:
On 8/19/07, Luca Tettamanti <[EMAIL PROTECTED]> wrote:
+static uint64_t qemu_next_deadline(void) {
+uint64_t nearest_delta_us = ULLONG_MAX;
+uint64_t vmdelta_us;
Hum, I introduced a bug here... those vars should be signed.
On the overhead introduced: how do you measure it?
Run a 100Hz guest, measure cpu usage using something accurate like
cyclesoak, with and without dynticks, with and without kvm.
Ok, here I've measured the CPU usage on the host when running an idle
guest.
[..snip the numbers..]
After briefly looking at the cyclesoak it indeed looks like it does
the right thing, but considering the limitations of user-space only
approach it introduces some (sometimes really unwanted) variables
into the system, those can, and i guess will, influence things.
The upshot is this - if you have used any standard utility (iostat,
top - basically anything /proc/stat based) the accounting has a fair
chance of being inaccurate. If cyclesoak is what you have used then
the results should be better, but still i would be worried about
them.
In conclusion until kernel native accounting is fixed your best bet
is to use: http://www.boblycat.org/~malc/apc/
--
vale
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Il Sun, Aug 19, 2007 at 10:31:26PM +0300, Avi Kivity ha scritto:
> Luca wrote:
> > On 8/19/07, Luca Tettamanti <[EMAIL PROTECTED]> wrote:
> >
> >> +static uint64_t qemu_next_deadline(void) {
> >> +uint64_t nearest_delta_us = ULLONG_MAX;
> >> +uint64_t vmdelta_us;
> >>
> >
> > Hum, I introduced a bug here... those vars should be signed.
> >
> > On the overhead introduced: how do you measure it?
> >
> >
>
> Run a 100Hz guest, measure cpu usage using something accurate like
> cyclesoak, with and without dynticks, with and without kvm.
Ok, here I've measured the CPU usage on the host when running an idle
guest.
At 100Hz
QEMU
hpet4.8%
dynticks5.1%
Note: I've taken the mean over a period of 20 secs, but the difference
between hpet and dynticks is well inside the variability of the test.
KVM
hpet2.2%
dynticks1.0%
Hum... here the numbers jumps a bit, but dynticks is always below hpet.
At 1000Hz:
QEMU
hpet5.5%
dynticks 11.7%
KVM
hpet3.4%
dynticks7.3%
No surprises here, you can see the additional 1k syscalls per second. On
the bright side, keep in mind that with a tickless guest and dynticks
I've seen as little as 50-60 timer ticks per second.
Hackbench (hackbench -pipe 50) inside the guest:
QEMU: impossible to measure, the variance of the results is much bigger
than difference between dynticks and hpet.
KVM:
Around 0.8s slower in case on dynticks; variance of the results is
about 0.3s in both cases.
Luca
--
"Chi parla in tono cortese, ma continua a prepararsi, potra` andare avanti;
chi parla in tono bellicoso e avanza rapidamente dovra` ritirarsi"
Sun Tzu -- L'arte della guerra
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Luca wrote:
> On 8/19/07, Luca Tettamanti <[EMAIL PROTECTED]> wrote:
>
>> +static uint64_t qemu_next_deadline(void) {
>> +uint64_t nearest_delta_us = ULLONG_MAX;
>> +uint64_t vmdelta_us;
>>
>
> Hum, I introduced a bug here... those vars should be signed.
>
> On the overhead introduced: how do you measure it?
>
>
Run a 100Hz guest, measure cpu usage using something accurate like
cyclesoak, with and without dynticks, with and without kvm.
--
Do not meddle in the internals of kernels, for they are subtle and quick to
panic.
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
On 8/19/07, Luca Tettamanti <[EMAIL PROTECTED]> wrote:
> +static uint64_t qemu_next_deadline(void) {
> +uint64_t nearest_delta_us = ULLONG_MAX;
> +uint64_t vmdelta_us;
Hum, I introduced a bug here... those vars should be signed.
On the overhead introduced: how do you measure it?
Luca
Re: [Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Avi Kivity wrote: > >>>In this case the dyn-tick minimum res will be 1msec. I believe it should > >>>work ok since this is the case without any dyn-tick. > >>> > >>Actually minimum resolution depends on host HZ setting, but - yes - > >>essentially you have the same behaviour of the "unix" timer, plus the > >>overhead of reprogramming the timer. > >> > > > >Is this significant? At a high guest HZ, this is could be quite a lot > >of additional syscalls right? > > > At HZ=1000, this adds a small multiple of 1000 syscalls, which is a > fairly small overhead. Small, but maybe measurable. That overhead could be removed if the dyn-tick code were to predictively set the host timer into a repeating mode when guests do actually require a regular tick. I'm thinking when it detects it needed a tick a small number of times in a row, with the same interval, it could set the host timer to trigger repeatedly at that interval. Then it wouldn't need to reprogram if that stayed the same (except maybe to correct for drift?) If a tick then _wasn't_ required for that interval (i.e. it was required for less, more, or not at all), then it would have to reprogram the host timer. If it really mattered, it wouldn't have to reprogram the host timer when the next required tick is further in the future or not at all; it would simply be a redundant SIGALRM. In weird cases that's worthwhile, but I suspect it generally isn't. -- Jamie
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Anthony Liguori wrote: I think this is a really nice and important patch set. Just a couple things: On Sun, 2007-08-19 at 00:02 +0200, Luca Tettamanti wrote: In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick. Actually minimum resolution depends on host HZ setting, but - yes - essentially you have the same behaviour of the "unix" timer, plus the overhead of reprogramming the timer. Is this significant? At a high guest HZ, this is could be quite a lot of additional syscalls right? At HZ=1000, this adds a small multiple of 1000 syscalls, which is a fairly small overhead. At HZ>1000 (only possible with a dyntick guest), the current implementation doesn't cope at all, so we can't compare overhead. -- error compiling committee.c: too many arguments to function
[Qemu-devel] RE: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure -take2
>I think this is a really nice and important patch set. Just a couple >things: > >On Sun, 2007-08-19 at 00:02 +0200, Luca Tettamanti wrote: > >> > In this case the dyn-tick minimum res will be 1msec. I believe it >should >> > work ok since this is the case without any dyn-tick. >> >> Actually minimum resolution depends on host HZ setting, but - yes - >> essentially you have the same behaviour of the "unix" timer, plus the >> overhead of reprogramming the timer. > >Is this significant? At a high guest HZ, this is could be quite a lot >of additional syscalls right? > I believe it's no significant since without dyn-tick the guest will get the same amount of signals so the overhead is doubling the syscalls (not a magnitude bigger). On the other size guests with low HZ and linux guests with dyn-tick will enojy from lesser syscalls.
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
I think this is a really nice and important patch set. Just a couple
things:
On Sun, 2007-08-19 at 00:02 +0200, Luca Tettamanti wrote:
> > In this case the dyn-tick minimum res will be 1msec. I believe it should
> > work ok since this is the case without any dyn-tick.
>
> Actually minimum resolution depends on host HZ setting, but - yes -
> essentially you have the same behaviour of the "unix" timer, plus the
> overhead of reprogramming the timer.
Is this significant? At a high guest HZ, this is could be quite a lot
of additional syscalls right?
> Add support for dynamic ticks.
>
> If the the dynticks alarm timer is used qemu does not attempt to generate
> SIGALRM at a constant rate. Rather, the system timer is set to generate
> SIGALRM
> only when it is needed. Dynticks timer reduces the number of SIGALRMs sent to
> idle dynamic-ticked guests.
> Original patch from Dan Kenigsberg <[EMAIL PROTECTED]>
>
> Signed-off-by: Luca Tettamanti <[EMAIL PROTECTED]>
>
> ---
> vl.c | 178
> +++
> 1 file changed, 170 insertions(+), 8 deletions(-)
>
> Index: qemu/vl.c
> ===
> --- qemu.orig/vl.c2007-08-18 23:23:47.0 +0200
> +++ qemu/vl.c 2007-08-18 23:23:53.0 +0200
> @@ -784,12 +784,31 @@
>
> struct qemu_alarm_timer {
> char const *name;
> +unsigned int flags;
>
> int (*start)(struct qemu_alarm_timer *t);
> void (*stop)(struct qemu_alarm_timer *t);
> +void (*rearm)(struct qemu_alarm_timer *t);
> void *priv;
> };
>
> +#define ALARM_FLAG_DYNTICKS 0x1
> +
> +static inline int alarm_has_dynticks(struct qemu_alarm_timer *t)
> +{
> +return t->flags & ALARM_FLAG_DYNTICKS;
> +}
> +
> +static void qemu_rearm_alarm_timer(struct qemu_alarm_timer *t) {
The '{' should be on the next line.
The rest looks fine.
Regards,
Anthony Liguori
> +if (!alarm_has_dynticks(t))
> +return;
> +
> +t->rearm(t);
> +}
> +
> +/* TODO: MIN_TIMER_REARM_US should be optimized */
> +#define MIN_TIMER_REARM_US 250
> +
> static struct qemu_alarm_timer *alarm_timer;
>
> #ifdef _WIN32
> @@ -802,12 +821,17 @@
>
> static int win32_start_timer(struct qemu_alarm_timer *t);
> static void win32_stop_timer(struct qemu_alarm_timer *t);
> +static void win32_rearm_timer(struct qemu_alarm_timer *t);
>
> #else
>
> static int unix_start_timer(struct qemu_alarm_timer *t);
> static void unix_stop_timer(struct qemu_alarm_timer *t);
>
> +static int dynticks_start_timer(struct qemu_alarm_timer *t);
> +static void dynticks_stop_timer(struct qemu_alarm_timer *t);
> +static void dynticks_rearm_timer(struct qemu_alarm_timer *t);
> +
> #ifdef __linux__
>
> static int hpet_start_timer(struct qemu_alarm_timer *t);
> @@ -816,21 +840,23 @@
> static int rtc_start_timer(struct qemu_alarm_timer *t);
> static void rtc_stop_timer(struct qemu_alarm_timer *t);
>
> -#endif
> +#endif /* __linux__ */
>
> #endif /* _WIN32 */
>
> static struct qemu_alarm_timer alarm_timers[] = {
> +#ifndef _WIN32
> +{"dynticks", ALARM_FLAG_DYNTICKS, dynticks_start_timer,
> dynticks_stop_timer, dynticks_rearm_timer, NULL},
> #ifdef __linux__
> /* HPET - if available - is preferred */
> -{"hpet", hpet_start_timer, hpet_stop_timer, NULL},
> +{"hpet", 0, hpet_start_timer, hpet_stop_timer, NULL, NULL},
> /* ...otherwise try RTC */
> -{"rtc", rtc_start_timer, rtc_stop_timer, NULL},
> +{"rtc", 0, rtc_start_timer, rtc_stop_timer, NULL, NULL},
> #endif
> -#ifndef _WIN32
> -{"unix", unix_start_timer, unix_stop_timer, NULL},
> +{"unix", 0, unix_start_timer, unix_stop_timer, NULL, NULL},
> #else
> -{"win32", win32_start_timer, win32_stop_timer, &alarm_win32_data},
> +{"dynticks", ALARM_FLAG_DYNTICKS, win32_start_timer, win32_stop_timer,
> win32_rearm_timer, &alarm_win32_data},
> +{"win32", 0, win32_start_timer, win32_stop_timer, NULL,
> &alarm_win32_data},
> #endif
> {NULL, }
> };
> @@ -949,6 +975,8 @@
> }
> pt = &t->next;
> }
> +
> +qemu_rearm_alarm_timer(alarm_timer);
> }
>
> /* modify the current timer so that it will be fired when current_time
> @@ -1008,6 +1036,7 @@
> /* run the callback (the timer list can be modified) */
> ts->cb(ts->opaque);
> }
> +qemu_rearm_alarm_timer(alarm_timer);
> }
>
> int64_t qemu_get_clock(QEMUClock *clock)
> @@ -1115,7 +1144,8 @@
> last_clock = ti;
> }
> #endif
> -if (qemu_timer_expired(active_timers[QEMU_TIMER_VIRTUAL],
> +if (alarm_has_dynticks(alarm_timer) ||
> +qemu_timer_expired(active_timers[QEMU_TIMER_VIRTUAL],
> qemu_get_clock(vm_clock)) ||
> qemu_timer_expired(active_timers[QEMU_TIMER_REALTIME],
> qemu_get_clock(rt_clock))) {
> @@ -1136,6 +1166,27 @@
> }
> }
>
> +static uint64_t qemu_next_deadline(void) {
> +
[Qemu-devel] Re: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
Dor Laor ha scritto:
> >> Hello,
> >> in reply to this mail I will send a serie of 4 patches that cleans up
> >and
> >> expands the alarm timer handling in QEMU. Patches have been rebased
> on
> >QEMU
> >> CVS.
> >>
> >> Patch 1 is mostly a cleanup of the existing code; instead of having
> >multiple
> >> #ifdefs to handle different timers scattered all over the code I've
> >created a
> >> modular infrastructure where each timer type is self-contained and
> >generic code
> >> is more readable. The resulting code is functionally equivalent to
> the
> >old one.
> >>
> >> Patch 2 implements the "-clock" command line option proposed by
> Daniel
> >Berrange
> >> and Avi Kivity. By default QEMU tries RTC and then falls back to unix
> >timer;
> >> user can override the order of the timer through this options. Syntax
> >is pretty
> >> simple: -clock timer1,timer2,etc. (QEMU will pick the first one that
> >works).
> >>
> >> Patch 3 adds support for HPET under Linux (which is basically my old
> >patch). As
> >> suggested HPET takes precedence over other timers, but of course this
> >can be
> >> overridden.
> >>
> >> Patch 4 introduces "dynticks" timer source; patch is mostly based on
> >the work
> >> Dan Kenigsberg. dynticks is now the default alarm timer.
> >
> >Why do you guard dynticks with #ifdef? Is there any reason why you
> >wouldn't want to use dynticks?
>
> I think too that it's should be the default.
> There is no regression in performance nor behaviour with this option.
Ok, I've updated the patch. It was pretty easy to implement the same
feature for win32 (slightly tested inside a winxp VM).
> We didn't test qemu dyn-tick with kernels that don't have
> high-res-timer+dyn-tick.
I did ;)
> In this case the dyn-tick minimum res will be 1msec. I believe it should
> work ok since this is the case without any dyn-tick.
Actually minimum resolution depends on host HZ setting, but - yes -
essentially you have the same behaviour of the "unix" timer, plus the
overhead of reprogramming the timer.
Add support for dynamic ticks.
If the the dynticks alarm timer is used qemu does not attempt to generate
SIGALRM at a constant rate. Rather, the system timer is set to generate SIGALRM
only when it is needed. Dynticks timer reduces the number of SIGALRMs sent to
idle dynamic-ticked guests.
Original patch from Dan Kenigsberg <[EMAIL PROTECTED]>
Signed-off-by: Luca Tettamanti <[EMAIL PROTECTED]>
---
vl.c | 178 +++
1 file changed, 170 insertions(+), 8 deletions(-)
Index: qemu/vl.c
===
--- qemu.orig/vl.c 2007-08-18 23:23:47.0 +0200
+++ qemu/vl.c 2007-08-18 23:23:53.0 +0200
@@ -784,12 +784,31 @@
struct qemu_alarm_timer {
char const *name;
+unsigned int flags;
int (*start)(struct qemu_alarm_timer *t);
void (*stop)(struct qemu_alarm_timer *t);
+void (*rearm)(struct qemu_alarm_timer *t);
void *priv;
};
+#define ALARM_FLAG_DYNTICKS 0x1
+
+static inline int alarm_has_dynticks(struct qemu_alarm_timer *t)
+{
+return t->flags & ALARM_FLAG_DYNTICKS;
+}
+
+static void qemu_rearm_alarm_timer(struct qemu_alarm_timer *t) {
+if (!alarm_has_dynticks(t))
+return;
+
+t->rearm(t);
+}
+
+/* TODO: MIN_TIMER_REARM_US should be optimized */
+#define MIN_TIMER_REARM_US 250
+
static struct qemu_alarm_timer *alarm_timer;
#ifdef _WIN32
@@ -802,12 +821,17 @@
static int win32_start_timer(struct qemu_alarm_timer *t);
static void win32_stop_timer(struct qemu_alarm_timer *t);
+static void win32_rearm_timer(struct qemu_alarm_timer *t);
#else
static int unix_start_timer(struct qemu_alarm_timer *t);
static void unix_stop_timer(struct qemu_alarm_timer *t);
+static int dynticks_start_timer(struct qemu_alarm_timer *t);
+static void dynticks_stop_timer(struct qemu_alarm_timer *t);
+static void dynticks_rearm_timer(struct qemu_alarm_timer *t);
+
#ifdef __linux__
static int hpet_start_timer(struct qemu_alarm_timer *t);
@@ -816,21 +840,23 @@
static int rtc_start_timer(struct qemu_alarm_timer *t);
static void rtc_stop_timer(struct qemu_alarm_timer *t);
-#endif
+#endif /* __linux__ */
#endif /* _WIN32 */
static struct qemu_alarm_timer alarm_timers[] = {
+#ifndef _WIN32
+{"dynticks", ALARM_FLAG_DYNTICKS, dynticks_start_timer,
dynticks_stop_timer, dynticks_rearm_timer, NULL},
#ifdef __linux__
/* HPET - if available - is preferred */
-{"hpet", hpet_start_timer, hpet_stop_timer, NULL},
+{"hpet", 0, hpet_start_timer, hpet_stop_timer, NULL, NULL},
/* ...otherwise try RTC */
-{"rtc", rtc_start_timer, rtc_stop_timer, NULL},
+{"rtc", 0, rtc_start_timer, rtc_stop_timer, NULL, NULL},
#endif
-#ifndef _WIN32
-{"unix", unix_start_timer, unix_stop_timer, NULL},
+{"unix", 0, unix_start_timer, unix_stop_timer, NULL, NULL},
#else
-{"win32", win32_start_timer, win32_stop_timer, &alarm_win32_data
[Qemu-devel] RE: [kvm-devel] [PATCH 0/4] Rework alarm timer infrastrucure - take2
>> Hello, >> in reply to this mail I will send a serie of 4 patches that cleans up >and >> expands the alarm timer handling in QEMU. Patches have been rebased on >QEMU >> CVS. >> >> Patch 1 is mostly a cleanup of the existing code; instead of having >multiple >> #ifdefs to handle different timers scattered all over the code I've >created a >> modular infrastructure where each timer type is self-contained and >generic code >> is more readable. The resulting code is functionally equivalent to the >old one. >> >> Patch 2 implements the "-clock" command line option proposed by Daniel >Berrange >> and Avi Kivity. By default QEMU tries RTC and then falls back to unix >timer; >> user can override the order of the timer through this options. Syntax >is pretty >> simple: -clock timer1,timer2,etc. (QEMU will pick the first one that >works). >> >> Patch 3 adds support for HPET under Linux (which is basically my old >patch). As >> suggested HPET takes precedence over other timers, but of course this >can be >> overridden. >> >> Patch 4 introduces "dynticks" timer source; patch is mostly based on >the work >> Dan Kenigsberg. dynticks is now the default alarm timer. > >Why do you guard dynticks with #ifdef? Is there any reason why you >wouldn't want to use dynticks? I think too that it's should be the default. There is no regression in performance nor behaviour with this option. We didn't test qemu dyn-tick with kernels that don't have high-res-timer+dyn-tick. In this case the dyn-tick minimum res will be 1msec. I believe it should work ok since this is the case without any dyn-tick. So, I join Anthony's vote. > >Regards, > >Anthony Liguori > >> Luca > > >--- - >- >This SF.net email is sponsored by: Splunk Inc. >Still grepping through log files to find problems? Stop. >Now Search log events and configuration files using AJAX and a browser. >Download your FREE copy of Splunk now >> http://get.splunk.com/ >___ >kvm-devel mailing list >[EMAIL PROTECTED] >https://lists.sourceforge.net/lists/listinfo/kvm-devel
