Re: High interrupt rate
On Monday 08 August 2011 21:30:41 b. f. wrote: I'll wait for your views on those before disabling polling on the kernel and hz=100. It looks like your interrupt rate, while probably higher than needed, is not unexpectedly high for your configuration. But you can lower it if you want to do so. You are using a system before the introduction of the new eventtimer code. If you use 9.x, that has the new code and some other timer-related improvements, and you are not performing polling, then you can achieve a large reduction in the number of timer interrupts when the system isn't busy. You can still achieve a reduction on 8.x, but the reduction usually won't be as large as on 9.x under similar conditions. To reduce timer interrupts on an idle system running 8.x or 9.x, if you do not need to poll (most systems do not), remove DEVICE_POLLING from your kernel, and lower kern.hz to a suitable value -- 100 or 250, for example. For many workloads, a lower value is not only adequate, but may also be better in some ways. Also, you may want to consider using your TSC as the system timecounter, because it is usually more efficient to do so. This may not work for SMP, because if there are multiple TSCs on your system, they may not be synchronized. In 9.x, there is a test for synchronization, and the TSCs are preferred to the ACPI-safe timer if they satisfy this test and meet some other requirements. In 8.x, the user has to tell the system that it is safe to use the TSCs by adding: kern.timecounter.smp_tsc=1 to /boot/loader.conf. If you are not putting your cores into the C3 state, then you could try setting this via the loader command line, booting, and then seeing if the kern.timecounter.tc.TSC.quality is positive, kern.timecounter.hardware is TSC, and everything is working as expected. If the results are satisfactory, then you could add the above entry to /boot/loader.conf. But it would be better to do this on 9.x, where there are some added safeguards. b. b.; Something really odd happened. After I sent you the data, while waiting for your reply, I changed Lusca cache to use 64M ram instead of the 256M it had. It was 1/8th of ram so I just decided to give it less. Well, I swear to you this was the ONLY thing I did!. Since then, the system has been running at around 97% idle 98% of the time! During load hours, there are only short(1s) spikes of 75%ish idle, far from each other. And web performance is actually a little better! And the overall response of the system improved. That's why I waited a couple of days to reply so I could confirm this behavior. I don't know. Maybe with more ram, lusca was spawning to many threads and thus loading the CPU but this is just a guess. I will take lusca memory back to 256 for the sake of checking but I want to find out if this new found estability is there to stay so I'll wait a little longer to do that. Your suggestions will be kept handy just in case. Thanks for everything. -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since 2.2.8 [not Pro-Audio YET!!] (99% winblows FREE) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: High interrupt rate
On 8/7/11, Mario Lobo l...@bsd.com.br wrote: On Sunday 07 August 2011 18:34:27 b. f. wrote: I know 75% idle is not bad but this machine, when not under load on a saturday night like today, used to be at around 98% idle 99% of the time. Now its is at 72% idle 99.9% of the time. It has been like this all day. The only things with a high interrupt rate are cpu0: timer 46922025 2000 cpu1: timer 46918117 1999 What could be causing this? I don't know that 2 timer interrupts per-cpu, per kern.hz, is altogether unexpected for some configurations, under some conditions. What happens if you boot with kern.hz=100 in /boot/loader.conf, or set via the loader command line? What happens if you remove the DEVICE_POLLING option from your kernel (and _not_ just disable polling per-device)? What is the output from sysctl kern.timecounter kern.eventtimer? b. Thanks b. ! [~]sysctl kern.timecounter kern.timecounter.tick: 1 kern.timecounter.choice: TSC(-100) ACPI-safe(850) i8254(0) dummy(-100) kern.timecounter.hardware: ACPI-safe kern.timecounter.stepwarnings: 0 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.i8254.counter: 39201 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.ACPI-safe.mask: 16777215 kern.timecounter.tc.ACPI-safe.counter: 1055460 kern.timecounter.tc.ACPI-safe.frequency: 3579545 kern.timecounter.tc.ACPI-safe.quality: 850 kern.timecounter.tc.TSC.mask: 4294967295 kern.timecounter.tc.TSC.counter: 1200011080 kern.timecounter.tc.TSC.frequency: 1995401152 kern.timecounter.tc.TSC.quality: -100 kern.timecounter.smp_tsc: 0 kern.timecounter.invariant_tsc: 1 [~]sysctl kern.hz kern.hz: 1000 [~]sysctl kern.eventtimer sysctl: unknown oid 'kern.eventtimer' I'll wait for your views on those before disabling polling on the kernel and hz=100. It looks like your interrupt rate, while probably higher than needed, is not unexpectedly high for your configuration. But you can lower it if you want to do so. You are using a system before the introduction of the new eventtimer code. If you use 9.x, that has the new code and some other timer-related improvements, and you are not performing polling, then you can achieve a large reduction in the number of timer interrupts when the system isn't busy. You can still achieve a reduction on 8.x, but the reduction usually won't be as large as on 9.x under similar conditions. To reduce timer interrupts on an idle system running 8.x or 9.x, if you do not need to poll (most systems do not), remove DEVICE_POLLING from your kernel, and lower kern.hz to a suitable value -- 100 or 250, for example. For many workloads, a lower value is not only adequate, but may also be better in some ways. Also, you may want to consider using your TSC as the system timecounter, because it is usually more efficient to do so. This may not work for SMP, because if there are multiple TSCs on your system, they may not be synchronized. In 9.x, there is a test for synchronization, and the TSCs are preferred to the ACPI-safe timer if they satisfy this test and meet some other requirements. In 8.x, the user has to tell the system that it is safe to use the TSCs by adding: kern.timecounter.smp_tsc=1 to /boot/loader.conf. If you are not putting your cores into the C3 state, then you could try setting this via the loader command line, booting, and then seeing if the kern.timecounter.tc.TSC.quality is positive, kern.timecounter.hardware is TSC, and everything is working as expected. If the results are satisfactory, then you could add the above entry to /boot/loader.conf. But it would be better to do this on 9.x, where there are some added safeguards. b. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: High interrupt rate
I know 75% idle is not bad but this machine, when not under load on a saturday night like today, used to be at around 98% idle 99% of the time. Now its is at 72% idle 99.9% of the time. It has been like this all day. The only things with a high interrupt rate are cpu0: timer 46922025 2000 cpu1: timer 46918117 1999 What could be causing this? I don't know that 2 timer interrupts per-cpu, per kern.hz, is altogether unexpected for some configurations, under some conditions. What happens if you boot with kern.hz=100 in /boot/loader.conf, or set via the loader command line? What happens if you remove the DEVICE_POLLING option from your kernel (and _not_ just disable polling per-device)? What is the output from sysctl kern.timecounter kern.eventtimer? b. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: High interrupt rate
On Sunday 07 August 2011 18:34:27 b. f. wrote: I know 75% idle is not bad but this machine, when not under load on a saturday night like today, used to be at around 98% idle 99% of the time. Now its is at 72% idle 99.9% of the time. It has been like this all day. The only things with a high interrupt rate are cpu0: timer 46922025 2000 cpu1: timer 46918117 1999 What could be causing this? I don't know that 2 timer interrupts per-cpu, per kern.hz, is altogether unexpected for some configurations, under some conditions. What happens if you boot with kern.hz=100 in /boot/loader.conf, or set via the loader command line? What happens if you remove the DEVICE_POLLING option from your kernel (and _not_ just disable polling per-device)? What is the output from sysctl kern.timecounter kern.eventtimer? b. Thanks b. ! [~]sysctl kern.timecounter kern.timecounter.tick: 1 kern.timecounter.choice: TSC(-100) ACPI-safe(850) i8254(0) dummy(-100) kern.timecounter.hardware: ACPI-safe kern.timecounter.stepwarnings: 0 kern.timecounter.tc.i8254.mask: 65535 kern.timecounter.tc.i8254.counter: 39201 kern.timecounter.tc.i8254.frequency: 1193182 kern.timecounter.tc.i8254.quality: 0 kern.timecounter.tc.ACPI-safe.mask: 16777215 kern.timecounter.tc.ACPI-safe.counter: 1055460 kern.timecounter.tc.ACPI-safe.frequency: 3579545 kern.timecounter.tc.ACPI-safe.quality: 850 kern.timecounter.tc.TSC.mask: 4294967295 kern.timecounter.tc.TSC.counter: 1200011080 kern.timecounter.tc.TSC.frequency: 1995401152 kern.timecounter.tc.TSC.quality: -100 kern.timecounter.smp_tsc: 0 kern.timecounter.invariant_tsc: 1 [~]sysctl kern.hz kern.hz: 1000 [~]sysctl kern.eventtimer sysctl: unknown oid 'kern.eventtimer' I'll wait for your views on those before disabling polling on the kernel and hz=100. Thanks again. -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since 2.2.8 [not Pro-Audio YET!!] (99% winblows FREE) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
High interrupt rate
Hi there; My system is a FreeBSD 8.2-STABLE #0: Wed Jul 13 17:32:44 BRT 2011 i386 I have 4 nics. sis0, rl0, vr0 and dc0. The last three are in polling mode (which I did to see if it would decrease the int amount). The int rate for sis0 is low. vmstat -i reports: interrupt total rate irq6: fdc0 1 0 irq14: ata0 163039 6 irq15: ata1 29 0 irq19: sis0 vr0 688525 29 irq20: ohci0 27 0 irq23: ehci0 2 0 cpu0: timer 46922025 2000 cpu1: timer 46918117 1999 Total 94691765 4036 But top reports: last pid: 5163; load averages: 0.04, 0.09, 0.08 up 0+06:30:13 20:13:59 144 processes: 3 running, 121 sleeping, 20 waiting CPU: 0.0% user, 0.0% nice, 1.2% system, 24.0% interrupt, 74.8% idle Mem: 42M Active, 154M Inact, 116M Wired, 96K Cache, 112M Buf, 1667M Free Swap: 4000M Total, 4000M Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 11 root 2 171 ki31 0K16K CPU00 706:33 103.37% idle 12 root 20 -32- 0K 160K WAIT1 67:16 97.46% intr 0 root 10 -680 0K72K - 1 1:34 2.98% kernel Systat reports: 1 usersLoad 0.12 0.11 0.08 Aug 6 20:15 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 560965076 194524 9784 1706152 count All 1139006624 234149213732 pages Proc:Interrupts r p d s w Csw Trp Sys Int Sof Fltcow4042 total 73 6150 10 405 44 30k zfodfdc0 irq6 ozfod ata0 irq14 1.5%Sys 25.0%Intr 0.0%User 0.0%Nice 73.5%Idle%ozfod ata1 irq15 ||||||||||| daefr44 sis0 vr0 1 = prcfr ohci0 20 5 dtbuf totfr ehci0 23 Namei Name-cache Dir-cache60 desvn react 1999 cpu0: time Callshits %hits % 27121 numvn pdwak 1999 cpu1: time 943 frevn pdpgs intrn Disks ad0 ad1 cd0 pass0 118476 wire KB/t 0.00 0.00 0.00 0.00 43796 act tps 0 0 0 0 157764 inact MB/s 0.00 0.00 0.00 0.00 96 cache %busy 0 0 0 0 1706056 free 114880 buf I know 75% idle is not bad but this machine, when not under load on a saturday night like today, used to be at around 98% idle 99% of the time. Now its is at 72% idle 99.9% of the time. It has been like this all day. The only things with a high interrupt rate are cpu0: timer 46922025 2000 cpu1: timer 46918117 1999 What could be causing this? thanks, -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since 2.2.8 [not Pro-Audio YET!!] (99% winblows FREE) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Surprisingly high interrupt rate on idle machine's uhci controller on 7.1
Hello, While checking out the -P option for top in 7.1 I noticed that my desktop at home (which is currently idle) was spending nearly 100% of one core on interrupt: last pid: 5678; load averages: 0.00, 0.00, 0.00 59 processes: 1 running, 58 sleeping CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 1: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 90.3% interrupt, 9.7% idle So I looked at vmstat -i: interrupt total rate irq16: nvidia0 em* 5009831 88 irq18: uhci2 ehci+ 66595 1 irq19: uhci4++2855320585 50637 irq22: pcm0 213238 3 irq23: uhci3 ehci1 1 0 cpu0: timer112447860 1994 cpu1: timer112444767 1994 cpu3: timer109457852 1941 cpu2: timer109457830 1941 Total 3304418559 58602 The interrupt rate for uhci4 seems very high for an idle system. % dmesg | grep uhci4 uhci4: UHCI (generic) USB controller port 0xb400-0xb41f irq 19 at device 29.1 on pci0 uhci4: [GIANT-LOCKED] uhci4: [ITHREAD] usb5: UHCI (generic) USB controller on uhci4 % dmesg | grep usb5 usb5: UHCI (generic) USB controller on uhci4 usb5: USB revision 1.0 uhub6: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 on usb5 usb7: companion controllers, 2 ports each: usb4 usb5 usb6 % usbdevs -v ... Controller /dev/usb5: addr 1: full speed, self powered, config 1, UHCI root hub(0x), Intel(0x), rev 1.00 port 1 powered port 2 powered Nothing is plugged in there! The only USB devices I have plugged in are: addr 1: EHCI root hub, Intel addr 2: USB2.0 Hub, vendor 0x05e3 addr 3: Microsoft 2.4GHz Transceiver v4.0, Microsoft addr 1: UHCI root hub, Intel addr 2: Generic USB Hub, Chicony addr 3: PFU-65 USB Keyboard, Chicony The receiver for my wireless mouse, which is plugged into a USB hub (this is to fix a different problem where if the receiver is directly plugged into the machine it stops working and FreeBSD complains that it has disabled the USB controller!), and my Happy Hacking keyboard which sits behind its own built-in hub. The machine is: % uname -a FreeBSD tomo.jamesoff.net 7.1-RELEASE FreeBSD 7.1-RELEASE #4: Mon Jan 26 22:16:37 UTC 2009 r...@tomo.jamesoff.net:/usr/obj/usr/src/sys/GENERIC i386 The board is an ASUS P5Q3 with a Q6600 2.4GHz installed. If there is any other information I should supply or debugging I can do, please let me know. /JMS ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Is this a high interrupt rate for nics?
On Sunday, 8 May 2005 at 9:02:08 -0400, M. Parsons wrote: Freebsd-5.3 SMP Kernel. Polling enabled. bash-2.05b$ vmstat -i interrupt total rate irq5: ep02937064 1 irq11: ed0 298318862165 irq10: de0 276544892152 Are those normal for ed0 and de0? Compared to the ep0 nic of rate of 1 (although the ep0 nic is not used as much as the other two nics obviously). Depends on what you mean by normal. You'll get one interrupt per packet, and those rates are perfectly normal. If you're not transferring anything, you shouldn't be getting any interrupts. Greg -- When replying to this message, please copy the original recipients. If you don't, I may ignore the reply or reply to the original recipients. For more information, see http://www.lemis.com/questions.html See complete headers for address and phone numbers ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Is this a high interrupt rate for nics?
Quoting Greg 'groggy' Lehey [EMAIL PROTECTED]: On Sunday, 8 May 2005 at 9:02:08 -0400, M. Parsons wrote: Freebsd-5.3 SMP Kernel. Polling enabled. bash-2.05b$ vmstat -i interrupt total rate irq5: ep02937064 1 irq11: ed0 298318862165 irq10: de0 276544892152 Are those normal for ed0 and de0? Compared to the ep0 nic of rate of 1 (although the ep0 nic is not used as much as the other two nics obviously). Depends on what you mean by normal. You'll get one interrupt per packet, and those rates are perfectly normal. If you're not transferring anything, you shouldn't be getting any interrupts. These are actually a bit low in my experience for full-speed 100 Mbit/sec traffic... I'll go up to the 2000-3000 range at full 100 Mbit speeds. Ken ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Is this a high interrupt rate for nics?
Greg 'groggy' Lehey wrote: On Sunday, 8 May 2005 at 9:02:08 -0400, M. Parsons wrote: Freebsd-5.3 SMP Kernel. Polling enabled. bash-2.05b$ vmstat -i interrupt total rate irq5: ep02937064 1 irq11: ed0 298318862165 irq10: de0 276544892152 Are those normal for ed0 and de0? Compared to the ep0 nic of rate of 1 (although the ep0 nic is not used as much as the other two nics obviously). Depends on what you mean by normal. You'll get one interrupt per packet, and those rates are perfectly normal. If you're not transferring anything, you shouldn't be getting any interrupts. Ok, thats what I thought. Just seeing the difference between ep0 and the other two made me a little confused. And seeing as ed0 and de0 are used about 100x more often than the ep0, those numbers seem about right then. Thanks for the answer. Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Is this a high interrupt rate for nics?
Freebsd-5.3 SMP Kernel. Polling enabled. bash-2.05b$ vmstat -i interrupt total rate irq1: atkbd0 20 0 irq5: ep02937064 1 irq6: fdc010 0 irq8: rtc 231396610128 irq14: ata0 1218635 0 irq15: ata1 173656 0 irq11: ed0 298318862165 irq10: de0 276544892152 irq0: clk 1807792593 1000 Total 2618382342 1448 Are those normal for ed0 and de0? Compared to the ep0 nic of rate of 1 (although the ep0 nic is not used as much as the other two nics obviously). Thanks. Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 'top' showing high interrupt rate
Thanks for the reply's. I didn't see them on the list. Maybe I missed them, so I have pasted them back in here. By any chance, have you altered the system quantum (HZ), or enabled kern.polling? -Chuck Run systat and then type :vmstat (without quotes) to get the vmstat screen. The lower right will have interrupts broken down by device ... this should help. -Bill Moran Try `top -S` instead. This will show system processes as well. Also, use `systat -v` as previously mentioned to see interrupt counts. My guess is this box is a gateway/firewall and may be getting pounded on the network interfaces. Are you running ipfw as well? - Andy Farkas Thanks guys! I tried: sysctl kern.polling and got:- sysctl: unknown oid 'kern.polling' I haven't played around with the system quantum. I like the systat -- :vmstat and the systat -v command. Very nice! Here is a dump of what is going on. As you can see, it doesn't show very much:- -- 1 usersLoad 0.21 0.17 0.20 May 25 10:32 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act6320 78413460 9447616 count All 248001880 2328768 3408 pages zfod Interrupts Proc:r p d s wCsw Trp Sys Int Sof Fltcow 241 total 172099 241 112 7600 wireata0 irq14 8568 act ata1 irq15 0.2%Sys 59.7%Intr 0.6%User 0.0%Nice 39.5%Idl 5284 inact 4 dc0 irq10 |||||||||| 3348 cache fdc0 irq6 ++ 4268 free 9 sio1 irq3 daefr 100 clk irq0 Namei Name-cacheDir-cache prcfr 128 rtc irq8 Calls hits% hits% react 33 100 pdwake pdpgs Disks ad0 ad1 acd0 fd0 md0 intrn KB/t 0.00 0.00 0.00 0.00 0.00 6480 buf tps 0 0 0 0 0 3 dirtybuf MB/s 0.00 0.00 0.00 0.00 0.00 2348 desiredvnodes % busy0 0 0 0 0 2374 numvnodes 1160 freevnodes -- Yes, this PC is a gateway to the internet via a modem (see dc0 and sio1 showing activity). I have tried the ststat -v on 3 or 4 servers, and they all pretty much show that clk and rtc use 100/128. However, most of them run at ~15% interupt activity. Hmm, maybe it's 60% on this server, because it's only a Pentium 166. Anyone else have an insight on this? Hmm, I then tried killing natd, and lo-and-behold, interrupts fell to 0.5%!! Restarted natd and interrupts are still 0.5%! Here is the natd via top before and after:- 108 root2 0 464K 152K select 96:45 0.00% 0.00% natd 15462 root2 0 420K 272K select 0:00 0.00% 0.00% natd The only difference is the size and res. Anyone know of problems with natd? The server was acting ok, just reporting that the interrupt usage was running at 60%. I will keep an eye on this over the next few days. Cheers, Paul Hamilton -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Paul Hamilton Sent: Wednesday, 21 May 2003 10:38 PM To: Freebsd-Questions Subject: 'top' showing high interrupt rate Hi, I noticed that top shows a rather high interrupt rate, usually hovering around 60%! Here is a sample:- last pid: 2879; load averages: 0.34, 0.24, 0.18 up 1+17:26:56 17:18:10 36 processes: 2 running, 33 sleeping, 1 zombie CPU states: 0.6% user, 0.0% nice, 0.0% system, 59.8% interrupt, 39.6% idle Mem: 11M Active, 5072K Inact, 7988K Wired, 3552K Cache, 6480K Buf, 960K Free Swap: 128M Total, 4608K Used, 123M Free, 3% Inuse PID USERNAME PRI NICE SIZERES STATETIME WCPUCPU COMMAND 135 root 10 0 980K 528K nanslp 11:38 0.00% 0.00% cron 201 root2 0 2824K 1132K select 3:46 0.00% 0.00% ppp 108 root2 0 452K 208K select 0:29 0.00% 0.00% natd 127 root2 0 2760K 1704K select 0:14 0.00% 0.00% named 223 root2 0 2068K 908K select 0:10 0.00% 0.00% sshd 212 root2 0 2044K 876K select 0:10 0.00% 0.00% sshd snip pstat -T
