Re: Why is intr taking up so much cpu?
Doug, could you please show your timer configuration, part of devinfo -u that describes interrupts and top of the output of top -SPH (including the header) when high interrupt load strikes? P.S. I saw output of top -SH, but I have a reason to be curious about top -SPH. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Wed, 21 Jul 2010, Andriy Gapon wrote: Doug, could you please show your timer configuration, Nothing special in /boot/loader.conf, /etc/sysctl.conf, or my kernel. It's basically just GENERIC minus devices I don't have, plus the following: options DDB_CTF options VESA options GEOM_BDE device atapicam device sound device snd_hda Interestingly, I had a runaway intr thing again after watching a flash video, but this time it was hdac0, not swi:4. http://people.freebsd.org/~dougb/bad-dtrace-3-hdac.txt http://people.freebsd.org/~dougb/bad-dtrace-4-hdac.txt part of devinfo -u that describes interrupts Interrupt request lines: 0 (attimer0) 1 (atkbd0) 3 (root0) 4 (uart0) 5-7 (root0) 8 (atrtc0) 9 (acpi0) 10-11 (root0) 12 (psm0) 12 (psmcpnp0) 13 (root0) 14 (ata0) 15 (ata1) 16 (root0) 17 (wpi0) 18 (cbb0) 19 (root0) 20 (ehci0) 20 (uhci0) 20 (hpet0) 21 (uhci1) 22 (uhci2) 23 (uhci3) 256 (hdac0) and top of the output of top -SPH (including the header) when high interrupt load strikes? Will do next time, thanks! Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
on 21/07/2010 21:50 Doug Barton said the following: On Wed, 21 Jul 2010, Andriy Gapon wrote: Doug, could you please show your timer configuration, Nothing special in /boot/loader.conf, /etc/sysctl.conf, or my kernel. It's basically just GENERIC minus devices I don't have, plus the following: I didn't mean your manual tuning, I meant how the system is configured :-) E.g. the relevant sysctl tree. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Wed, 21 Jul 2010, Andriy Gapon wrote: I didn't mean your manual tuning, I meant how the system is configured :-) E.g. the relevant sysctl tree. Duh. :) Sorry. sysctl -a | grep timer kern.eventtimer.choice: LAPIC(500) HPET(450) HPET1(440) HPET2(440) i8254(100) RTC(0) kern.eventtimer.et.LAPIC.flags: 15 kern.eventtimer.et.LAPIC.frequency: 83223728 kern.eventtimer.et.LAPIC.quality: 500 kern.eventtimer.et.HPET.flags: 3 kern.eventtimer.et.HPET.frequency: 14318180 kern.eventtimer.et.HPET.quality: 450 kern.eventtimer.et.HPET1.flags: 3 kern.eventtimer.et.HPET1.frequency: 14318180 kern.eventtimer.et.HPET1.quality: 440 kern.eventtimer.et.HPET2.flags: 3 kern.eventtimer.et.HPET2.frequency: 14318180 kern.eventtimer.et.HPET2.quality: 440 kern.eventtimer.et.RTC.flags: 17 kern.eventtimer.et.RTC.frequency: 32768 kern.eventtimer.et.RTC.quality: 0 kern.eventtimer.et.i8254.flags: 1 kern.eventtimer.et.i8254.frequency: 1193182 kern.eventtimer.et.i8254.quality: 100 kern.eventtimer.timer2: HPET kern.eventtimer.timer1: LAPIC kern.eventtimer.singlemul: 2 net.inet.tcp.timer_race: 0 net.inet.tcp.per_cpu_timers: 0 machdep.acpi_timer_freq: 3579545 p1003_1b.timers: 200112 p1003_1b.delaytimer_max: 2147483647 p1003_1b.timer_max: 32 dev.acpi_timer.0.%desc: 24-bit timer at 3.579545MHz dev.acpi_timer.0.%driver: acpi_timer dev.acpi_timer.0.%location: unknown dev.acpi_timer.0.%pnpinfo: unknown dev.acpi_timer.0.%parent: acpi0 dev.attimer.0.%desc: AT timer dev.attimer.0.%driver: attimer dev.attimer.0.%location: handle=\_SB_.PCI0.ISAB.TMR_ dev.attimer.0.%pnpinfo: _HID=PNP0100 _UID=0 dev.attimer.0.%parent: acpi0 dev.pmtimer.0.%driver: pmtimer dev.pmtimer.0.%parent: isa0 -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sun, 18 Jul 2010, Dan Nelson wrote: You can also use dtrace to get a count of callouts and their time spent. Run this for a few seconds then hit ^C: Okey dokey, here you go: http://people.freebsd.org/~dougb/normal-dtrace.txt http://people.freebsd.org/~dougb/bad-dtrace.txt Thanks again, Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sun, 18 Jul 2010, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Ok, I thought I was going to be able to do this easily but I didn't realize that the numbers in the second column were thread ids, and I don't know how to correlate the clock thread tid with the backtrace. Can you give me a hint? :) Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Mon, Jul 19, 2010 at 11:05:26PM -0700, Doug Barton wrote: On Sun, 18 Jul 2010, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Ok, I thought I was going to be able to do this easily but I didn't realize that the numbers in the second column were thread ids, and I don't know how to correlate the clock thread tid with the backtrace. Can you give me a hint? :) It already printed the thread names, so no need. Unfortunately, the clock threads were running instead of blocking etc (I suspected that this would be a case), so procstat cannot get the backtrace. Another option is to do a backtrace from ddb. I cannot get much information from the dtrace snippets you posted in parallel. I can only see that some threads used msleep (?) with timeout a lot, and something at the address 0xc67bbe90 also raised a head. Can you manually lookup nearby symbol for 0xc67bbe90 ? pgp91DUQuoccc.pgp Description: PGP signature
Re: Why is intr taking up so much cpu?
In the last episode (Jul 19), Doug Barton said:
On Sun, 18 Jul 2010, Dan Nelson wrote:
You can also use dtrace to get a count of callouts and their time spent.
Run this for a few seconds then hit ^C:
Okey dokey, here you go:
http://people.freebsd.org/~dougb/normal-dtrace.txt
http://people.freebsd.org/~dougb/bad-dtrace.txt
I don't see any real difference between those two runs, so maybe it's not a
callout eating your CPU. How about running this for a few seconds, which
will print all the stack traces seen during the sampling period:
dtrace -n 'profile:::profile-276hz { @pc[stack()]=count(); }'
On an otherwise idle system, you should see most of the counts in cpu_idle,
with the remainder clustered in whatever code is eating your CPU.
--
Dan Nelson
dnel...@allantgroup.com
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sun, Jul 18, 2010 at 10:06:06PM -0700, Doug Barton wrote:
On 07/18/10 12:41, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote:
On 07/18/10 03:30, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote:
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the
processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4:
clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Most likely it is some callout handling. Just in case, do you have
console screensaver active ?
I assume you mean saver=yes in rc.conf, and the answer is no, I am not
using that. Usually I run xscreensaver, but at the time this happened I
was not. I do have DPMS enabled in my X config though.
Any suggestions on how to dig deeper on this? Are there any settings I
can twiddle to try and mitigate it?
When intr time starts accumulating again, try to do
procstat -kk intr process pid and correlate the clock thread tid
with the backtrace. Might be, it helps to guess what callouts are eating
the CPU.
Ok, file attached.
--
Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/
Computers are useless. They can only give you answers.
-- Pablo Picasso
PIDTID COMM TDNAME KSTACK
11 14 intr swi1: netisr 0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 15 intr swi4: clock mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 16 intr swi4: clock mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 17 intr swi3: vm
11 100014 intr swi6: Giant task mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100015 intr swi6: task queue mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100020 intr swi2: cambio mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100021 intr swi5: +
11 100022 intr irq9: acpi0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100023 intr irq16:
11 100024 intr irq256: hdac0mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100026 intr irq17: wpi0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100027 intr irq20: hpet0 uhc mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100032 intr irq21: uhci1
11 100037 intr irq22: uhci2 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100042 intr irq23: uhci3
11 100052 intr irq14: ata0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100053 intr irq15: ata1 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100055 intr irq1: atkbd0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100056 intr irq12: psm0
11 100057 intr swi0: uart
You should correlate the backtrace and the id of the cpu-consuming thread
(15 or 16, or both) and do periodic procstat -k to see which
functions are referenced most often.
Might be, suggested dtrace solution is easier.
pgpdw3vZqYxla.pgp
Description: PGP signature
Re: Why is intr taking up so much cpu?
I added options KDTRACE_HOOKS to my kernel config, built a new kernel,
and rebooted. I decided to try your script before things went sideways
so I'd have an idea of what to expect, and it didn't work:
dtrace: failed to initialize dtrace: DTrace device not available on
system
Is there something else I need to do to enable it?
Doug
On Sun, 18 Jul 2010, Dan Nelson wrote:
You can also use dtrace to get a count of callouts and their time spent.
Run this for a few seconds then hit ^C:
#! /usr/sbin/dtrace -s
/* #pragma D option quiet */
callout_execute:::callout_start
{
this-start = timestamp;
}
callout_execute:::callout_end
{
this-end = timestamp;
/* printf(%a %d\n,args[0]-c_func, this-end - this-start); */
@times[args[0]-c_func] = quantize(this-end - this-start);
/* @times[args[0]-c_func] = lquantize(this-end -
this-start,0,30,1); */
@counts[args[0]-c_func] = count();
}
END
{
printa(%a %...@u\n,@times);
printa(%a %...@u\n,@counts);
}
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there something else I need to do to enable it? You need to build the kernel with CTF. Try adding makeoptions WITH_CTF=yes to your config and rebuilding your kernel. There's a blurb in src/UPDATING about other ways to accomplish the same thing. -- Chris - http://twitter.com/chrisattack http://chrisattack.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there something else I need to do to enable it? You need to build the kernel with CTF. Try adding makeoptions WITH_CTF=yes to your config and rebuilding your kernel. There's a blurb in src/UPDATING about other ways to accomplish the same thing. Thanks for the suggestion, but no improvement. Doing: strings /boot/kernel/kernel | grep -i dtrace Shows lots of dtrace-related entries, unlike previous kernels built without the KDTRACE_HOOKS option, but same error with Dan's script. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Mon, Jul 19, 2010 at 07:33:01PM -0700, Doug Barton wrote: On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there something else I need to do to enable it? You need to build the kernel with CTF. Try adding makeoptions WITH_CTF=yes to your config and rebuilding your kernel. There's a blurb in src/UPDATING about other ways to accomplish the same thing. Thanks for the suggestion, but no improvement. Doing: strings /boot/kernel/kernel | grep -i dtrace Shows lots of dtrace-related entries, unlike previous kernels built without the KDTRACE_HOOKS option, but same error with Dan's script. Try a kldload dtraceall before running the script. Regards, Navdeep ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Tuesday 20 July 2010 04:33:01 Doug Barton wrote: On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there something else I need to do to enable it? You need to build the kernel with CTF. Try adding makeoptions WITH_CTF=yes to your config and rebuilding your kernel. There's a blurb in src/UPDATING about other ways to accomplish the same thing. Thanks for the suggestion, but no improvement. Doing: strings /boot/kernel/kernel | grep -i dtrace Shows lots of dtrace-related entries, unlike previous kernels built without the KDTRACE_HOOKS option, but same error with Dan's script. Just a stab in the dark, did you kldload dtraceall? KDTRACE_HOOKS just adds the needed linkage for the dtrace modules to work. Max ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Tue, 20 Jul 2010, Max Laier wrote: Just a stab in the dark, did you kldload dtraceall? KDTRACE_HOOKS just adds the needed linkage for the dtrace modules to work. No, I had not done that, in fact, I didn't even know I needed those modules. I use MODULES_OVERRIDE so I had to add dtrace, cyclic, and opensolaris to the list. In any case ... It's working now! :) I'm collecting some data for normal atm, then I'll try to get it into the situation where intr runs away, and I'll do the same thing again. Thanks Max and Chris, Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
In the last episode (Jul 19), Doug Barton said: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there something else I need to do to enable it? I think you also need WITH_CTF=yes , either in your kernel config or directly on the make commandline. The kernel config option should work, but if it doesn't, it's guaranteed to work on the commandline. http://wiki.freebsd.org/DTrace http://lists.freebsd.org/pipermail/freebsd-current/2010-April/016620.html -- Dan Nelson dnel...@allantgroup.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4: clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Doug
--
Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/
Computers are useless. They can only give you answers.
-- Pablo Picasso
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, Jul 17, 2010 at 10:21:28PM +0300, Kostik Belousov wrote: On Sat, Jul 17, 2010 at 12:10:26PM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like that. Given that there are other -current users who are also having problems (particularly with the nvidia drivers) I'm wondering if some sort of systemic debugging isn't in order here? Note that intr time most likely come from the interrupt threads chewing the CPU, not from the real interrupt handlers doing something, and definitely not due to the high interrupt rate, as your vmstat -i output already shown. I've noticed a few webpages to trigger lot of X11 related network traffic just by watching them even without any seeable content change, but CPU load on browser and especialy X process went high, but of course symptoms might be different with different drivers - I use mga myself. I never analysed it properly beacuse I'm using a quite old Xorg version, but I see the increase of traffic on the domain socket. I also noticed that recent firefox and seamonkey are doing lots of NFS traffic, so I was forced to switch ~/.mozilla to a local disk, where iostat still stays idle. But my OS is also not very recent, so I also never debugged this problem. Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. -- B.Walter be...@bwct.de http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote:
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4:
clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Most likely it is some callout handling. Just in case, do you have
console screensaver active ?
pgpnr7b4o3rZt.pgp
Description: PGP signature
Re: Why is intr taking up so much cpu?
On 07/18/10 03:30, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote:
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4:
clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Most likely it is some callout handling. Just in case, do you have
console screensaver active ?
I assume you mean saver=yes in rc.conf, and the answer is no, I am not
using that. Usually I run xscreensaver, but at the time this happened I
was not. I do have DPMS enabled in my X config though.
Any suggestions on how to dig deeper on this? Are there any settings I
can twiddle to try and mitigate it?
Doug
--
... and that's just a little bit of history repeating.
-- Propellerheads
Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote:
On 07/18/10 03:30, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote:
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4:
clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Most likely it is some callout handling. Just in case, do you have
console screensaver active ?
I assume you mean saver=yes in rc.conf, and the answer is no, I am not
using that. Usually I run xscreensaver, but at the time this happened I
was not. I do have DPMS enabled in my X config though.
Any suggestions on how to dig deeper on this? Are there any settings I
can twiddle to try and mitigate it?
When intr time starts accumulating again, try to do
procstat -kk intr process pid and correlate the clock thread tid
with the backtrace. Might be, it helps to guess what callouts are eating
the CPU.
pgpzAHoszwKlb.pgp
Description: PGP signature
Re: Why is intr taking up so much cpu?
On 07/18/10 12:41, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Will do, thanks! Doug -- ... and that's just a little bit of history repeating. -- Propellerheads Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
In the last episode (Jul 18), Doug Barton said:
On 07/18/10 12:41, Kostik Belousov wrote:
When intr time starts accumulating again, try to do
procstat -kk intr process pid and correlate the clock thread tid
with the backtrace. Might be, it helps to guess what callouts are eating
the CPU.
Will do, thanks!
You can also use dtrace to get a count of callouts and their time spent.
Run this for a few seconds then hit ^C:
#! /usr/sbin/dtrace -s
/* #pragma D option quiet */
callout_execute:::callout_start
{
this-start = timestamp;
}
callout_execute:::callout_end
{
this-end = timestamp;
/* printf(%a %d\n,args[0]-c_func, this-end - this-start); */
@times[args[0]-c_func] = quantize(this-end - this-start);
/* @times[args[0]-c_func] = lquantize(this-end -
this-start,0,30,1); */
@counts[args[0]-c_func] = count();
}
END
{
printa(%a %...@u\n,@times);
printa(%a %...@u\n,@counts);
}
--
Dan Nelson
dnel...@allantgroup.com
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
In the last episode (Jul 18), Dan Nelson said: In the last episode (Jul 18), Doug Barton said: On 07/18/10 12:41, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Will do, thanks! You can also use dtrace to get a count of callouts and their time spent. Run this for a few seconds then hit ^C: That may actually be too verbose (you'll get a histogram per callout). Try the ones at http://wiki.freebsd.org/DTrace/Examples instead. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On 07/18/10 12:41, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote:
On 07/18/10 03:30, Kostik Belousov wrote:
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote:
On Sat, 17 Jul 2010, Kostik Belousov wrote:
Run top in the mode where all system threads are shown separately
(e.g. top -HS seems to do it), then watch what thread eats the processor.
And the winner is!
11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4:
clock}
11 root21 -64- 0K 168K WAIT0 1:17 18.90% intr
The first is with -H, the second without.
Most likely it is some callout handling. Just in case, do you have
console screensaver active ?
I assume you mean saver=yes in rc.conf, and the answer is no, I am not
using that. Usually I run xscreensaver, but at the time this happened I
was not. I do have DPMS enabled in my X config though.
Any suggestions on how to dig deeper on this? Are there any settings I
can twiddle to try and mitigate it?
When intr time starts accumulating again, try to do
procstat -kk intr process pid and correlate the clock thread tid
with the backtrace. Might be, it helps to guess what callouts are eating
the CPU.
Ok, file attached.
--
Improve the effectiveness of your Internet presence with
a domain name makeover!http://SupersetSolutions.com/
Computers are useless. They can only give you answers.
-- Pablo Picasso
PIDTID COMM TDNAME KSTACK
11 14 intr swi1: netisr 0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 15 intr swi4: clock mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 16 intr swi4: clock mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 17 intr swi3: vm
11 100014 intr swi6: Giant task mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100015 intr swi6: task queue mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100020 intr swi2: cambio mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100021 intr swi5: +
11 100022 intr irq9: acpi0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100023 intr irq16:
11 100024 intr irq256: hdac0mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100026 intr irq17: wpi0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100027 intr irq20: hpet0 uhc mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100032 intr irq21: uhci1
11 100037 intr irq22: uhci2 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100042 intr irq23: uhci3
11 100052 intr irq14: ata0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100053 intr irq15: ata1 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100055 intr irq1: atkbd0 mi_switch+0x200
ithread_loop+0x1da fork_exit+0xb8 fork_trampoline+0x8
11 100056 intr irq12: psm0
11 100057 intr swi0: uart
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Why is intr taking up so much cpu?
This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat -i interrupt total rate irq1: atkbd0 10384 0 irq9: acpi05 0 irq14: ata0 153410 7 irq15: ata1 58 0 irq17: wpi0 534038 27 irq20: hpet0 uhci0+ 2496833129 irq22: uhci2 66485 3 cpu0:timer 19238037999 irq256: hdac0 189713 9 cpu1:timer 19236431999 Total 41925394 2178 Any suggestions? current (r210135), i386 smp. Dell C2D laptop. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, 17 Jul 2010, Rui Paulo wrote: On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat -i interrupt total rate irq1: atkbd0 10384 0 irq9: acpi05 0 irq14: ata0 153410 7 irq15: ata1 58 0 irq17: wpi0 534038 27 irq20: hpet0 uhci0+ 2496833129 irq22: uhci2 66485 3 cpu0:timer 19238037999 irq256: hdac0 189713 9 cpu1:timer 19236431999 Total 41925394 2178 Any suggestions? current (r210135), i386 smp. Dell C2D laptop. What's vmstat -i before the event happens? Here is the output after a clean boot: interrupt total rate irq1: atkbd0 424 4 irq9: acpi02 0 irq14: ata0 3266 30 irq15: ata1 58 0 irq17: wpi0 2012 18 irq20: hpet0 uhci0+13763129 irq22: uhci2 16 0 cpu0:timer105150991 irq256: hdac0 10 0 cpu1:timer103716978 Total 228417 2154 Thanks for the response, Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On 17 Jul 2010, at 19:04, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat -i interrupt total rate irq1: atkbd0 10384 0 irq9: acpi05 0 irq14: ata0 153410 7 irq15: ata1 58 0 irq17: wpi0 534038 27 irq20: hpet0 uhci0+ 2496833129 irq22: uhci2 66485 3 cpu0:timer 19238037999 irq256: hdac0 189713 9 cpu1:timer 19236431999 Total 41925394 2178 Any suggestions? current (r210135), i386 smp. Dell C2D laptop. What's vmstat -i before the event happens? Here is the output after a clean boot: interrupt total rate irq1: atkbd0 424 4 irq9: acpi02 0 irq14: ata0 3266 30 irq15: ata1 58 0 irq17: wpi0 2012 18 irq20: hpet0 uhci0+13763129 irq22: uhci2 16 0 cpu0:timer105150991 irq256: hdac0 10 0 cpu1:timer103716978 Total 228417 2154 Thanks for the response, This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. Regards, -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat -i interrupt total rate irq1: atkbd0 10384 0 irq9: acpi05 0 irq14: ata0 153410 7 irq15: ata1 58 0 irq17: wpi0 534038 27 irq20: hpet0 uhci0+ 2496833129 irq22: uhci2 66485 3 cpu0:timer 19238037999 irq256: hdac0 189713 9 cpu1:timer 19236431999 Total 41925394 2178 Any suggestions? current (r210135), i386 smp. Dell C2D laptop. What's vmstat -i before the event happens? Regards, -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like that. Given that there are other -current users who are also having problems (particularly with the nvidia drivers) I'm wondering if some sort of systemic debugging isn't in order here? Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On 17 Jul 2010, at 20:10, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like that. Given that there are other -current users who are also having problems (particularly with the nvidia drivers) I'm wondering if some sort of systemic debugging isn't in order here? You can try bisecting the faulty revision. Regards, -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, 17 Jul 2010, Rui Paulo wrote: You can try bisecting the faulty revision. The problem has been going on for months, the primary symptom for a long time was the nvidia driver, so I stopped using it for a while hoping that a solution would magically appear. As of the last 6 weeks or so the problem has started happening even without using the nvidia driver, and more users are reporting similar symptoms. So in short, no, I won't be doing that, as there is way too much history to slog back through at this point. What I would like to see is some sort of effort on the part of those who've made the changes to help debug what's wrong with them. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Why is intr taking up so much cpu?
On Sat, Jul 17, 2010 at 12:10:26PM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like that. Given that there are other -current users who are also having problems (particularly with the nvidia drivers) I'm wondering if some sort of systemic debugging isn't in order here? Note that intr time most likely come from the interrupt threads chewing the CPU, not from the real interrupt handlers doing something, and definitely not due to the high interrupt rate, as your vmstat -i output already shown. Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. pgpndi3E8dqD5.pgp Description: PGP signature
Re: Why is intr taking up so much cpu?
On Sat, 17 Jul 2010, Kostik Belousov wrote: Note that intr time most likely come from the interrupt threads chewing the CPU, not from the real interrupt handlers doing something, and definitely not due to the high interrupt rate, as your vmstat -i output already shown. Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. Ok, thanks, I'll definitely do that next time and report the results. Doug -- Improve the effectiveness of your Internet presence with a domain name makeover!http://SupersetSolutions.com/ Computers are useless. They can only give you answers. -- Pablo Picasso ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
