Re: CPU usage 100% but no process hogging CPU
Mario Lobo wrote: On Sunday 28 October 2007 10:38:36 Kris Kennaway wrote: Oh, you were going by the load average? That is not a measure of system performance, it only shows how many processes are running. Anyway, glad you resolved it to your satisfaction. Kris What would be the proper way to measure system performance/load then ? By measuring performance, i.e. how well it performs according to whatever criteria are important to you :) You can use load measurements like load average and CPU usage to focus your performance measurements but they are not themselves performance metrics. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: CPU usage 100% but no process hogging CPU
On Sunday 28 October 2007 10:38:36 Kris Kennaway wrote: > Oh, you were going by the load average? That is not a measure of system > performance, it only shows how many processes are running. > > Anyway, glad you resolved it to your satisfaction. > > Kris > What would be the proper way to measure system performance/load then ? -- Mario Lobo http://www.mallavoodoo.com.br FreeBSD since version 2.2.8 [not Pro-Audio YET!!] (99,7% winedows FREE) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: CPU usage 100% but no process hogging CPU
Gunther Mayer wrote: Kris Kennaway wrote: Gunther Mayer wrote: I don't see why my javavm, apache, postgres and/or radiusd would spawn such short lived processes. Come to think of it, I know radius might be doing just that, but how the heck would I go about finding out? top -H brings me no closer... Either increase the sample resolution or rule out other programs as the cause. Increasing sample resolution just increases the CPU usage of top itself but gives no more information about the real culprits :-( But I managed to solve my problem by reverting to and upgrading my SMP kernel. The system's back now with 15 minute load averages around 0.05 which is where it should be. Oh, you were going by the load average? That is not a measure of system performance, it only shows how many processes are running. Anyway, glad you resolved it to your satisfaction. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: CPU usage 100% but no process hogging CPU
Kris Kennaway wrote: Gunther Mayer wrote: I don't see why my javavm, apache, postgres and/or radiusd would spawn such short lived processes. Come to think of it, I know radius might be doing just that, but how the heck would I go about finding out? top -H brings me no closer... Either increase the sample resolution or rule out other programs as the cause. Increasing sample resolution just increases the CPU usage of top itself but gives no more information about the real culprits :-( But I managed to solve my problem by reverting to and upgrading my SMP kernel. The system's back now with 15 minute load averages around 0.05 which is where it should be. What really remains a puzzle is what caused the bottleneck on my system running the GENERIC single CPU system. I actually went back to my daily cron emails in which it states that load average at 5am increased from around 0 to 1.0 somewhere between day 3 and day 4 after rebooting into freebsd-update's erroneously downloaded GENERIC kernel. It stayed around 1.0 until this morning when I reverted and rebooted. On even closer inspection, before doing freebsd-update I had an uptime of 202 days on that box. Even then there are some days when load average was 1.0 at 5am, I think the real culprit is my radiusd which probably sometimes spawns threads that chow 100% CPU until killed somehow. Having only one CPU available would of course exacerbate such a problem. Seeing that performance is back 100% I think I will wait for 7.0 to supercharge my system's threads. Anyways, thanks for the help Kris. Gunther I see that my java is using no less than 26 threads, thread usage not showing up might well be the problem. Interesting point you make about libthr. I had no idea about the different threading options available on 6.x and did some reading up on it but there's very little official documentation or recommendations about it. Upon investigation it turns out that all my core daemons (httpd, postgres, radiusd and java) are linked against /lib/libpthread.so.2 which afaict after limited reading is what you refer to as libkse. Is that correct? Yes. So I'm tempted to use libmap.conf to switch to libthr for all instances of libpthread, though I'm put off by some very recent reports (http://roy.marples.name/node/332) that this can cause some nasty problems. Do you think that's cause for concern? That bug report makes no sense to me. If they are using libmap correctly then *all* libpthread references are remapped and if there are missing symbols they will cause failure from the dynamic linker when the process is first executed, not random crashes during operation. Would you mind sharing your libmap.conf and/or symlink setup with the list as well as your experiences with libthr? See libmap.conf(5). Then again, as a first step I should really get my SMP kernel back as a first step... That will surely help. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: CPU usage 100% but no process hogging CPU
Gunther Mayer wrote: Hi there, I'm having some capacity issues on the FreeBSD 6.2/Core 2 Duo/2GB RAM server that I manage. For quite a few days now it constantly shows load averages of around 1 and a CPU usage of around 100%. Yet summing up the CPU usage of the individual processes running I hardly ever get to more than 5%, regardless of how long I watch top. A snapshot of my top output looks like this: last pid: 96102; load averages: 1.28, 1.15, 1.06 up 22+08:33:16 13:55:03 122 processes: 2 running, 119 sleeping, 1 zombie CPU states: 67.3% user, 0.0% nice, 32.7% system, 0.0% interrupt, 0.0% idle Mem: 474M Active, 974M Inact, 186M Wired, 68M Cache, 213M Buf, 93M Free Swap: 4064M Total, 4064M Free PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 635 root1 1220 27304K 2644K select 656:38 1.27% syslog-ng 844 www20 200 411M 300M kserel 360:13 0.00% java 837 user1 3 200 29048K 5672K kserel 34:30 0.00% radiusd 788 pgsql 1 960 13516K 3824K select 10:03 0.00% postgres 785 pgsql 1 1150 120M 7436K select 9:02 0.00% postgres 787 pgsql 1 80 120M 41112K nanslp 5:15 0.00% postgres syslog-ng is quite busy as I use it to capture logs of more than 50 remote sites. I have lots of slow queries in my postgres logs that I think are related to this bottleneck, though unoptimised queries and an ever growing amount of data are more likely to take the blame for that. High disk I/O in this regard could explain the high system utilisation, however. I found out that I've been bitten by the freebsd-update bug (http://security.freebsd.org/advisories/FreeBSD-EN-07:05.freebsd-update.asc) which replaced my SMP kernel with a GENERIC one and I'm taking corrective action early tomorrow morning, but surely even with just a single CPU the load average should never be as high? Where are those phantom CPU hogging processes? A couple of points: 1) top -S will show what the kernel is doing, which may be relevant. 2) Because it only samples once a second (by default), top is bad for monitoring of any short-lived processes that may be using CPU for brief periods and then exiting. Don't know if you have any on this workload though. 3) In 6.x threaded applications do not generate CPU usage data in top. , i.e. java is probably using more than 0% of your CPU :) I think this is fixed in 7.0 and maybe also with libthr. Chances are you want to use libthr even in 6.x for performance reasons (libkse has attrocious performance). Use libmap.conf to switch the libraries. Kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: CPU usage 100% but no process hogging CPU
Gunther Mayer wrote: Hi there, I'm having some capacity issues on the FreeBSD 6.2/Core 2 Duo/2GB RAM server that I manage. For quite a few days now it constantly shows load averages of around 1 and a CPU usage of around 100%. Yet summing up the CPU usage of the individual processes running I hardly ever get to more than 5%, regardless of how long I watch top. A snapshot of my top output looks like this: last pid: 96102; load averages: 1.28, 1.15, 1.06 up 22+08:33:16 13:55:03 122 processes: 2 running, 119 sleeping, 1 zombie CPU states: 67.3% user, 0.0% nice, 32.7% system, 0.0% interrupt, 0.0% idle Mem: 474M Active, 974M Inact, 186M Wired, 68M Cache, 213M Buf, 93M Free Swap: 4064M Total, 4064M Free PID USERNAME THR PRI NICE SIZERES STATETIME WCPU COMMAND 635 root1 1220 27304K 2644K select 656:38 1.27% syslog-ng 844 www20 200 411M 300M kserel 360:13 0.00% java 837 user1 3 200 29048K 5672K kserel 34:30 0.00% radiusd 788 pgsql 1 960 13516K 3824K select 10:03 0.00% postgres 785 pgsql 1 1150 120M 7436K select 9:02 0.00% postgres 787 pgsql 1 80 120M 41112K nanslp 5:15 0.00% postgres syslog-ng is quite busy as I use it to capture logs of more than 50 remote sites. I have lots of slow queries in my postgres logs that I think are related to this bottleneck, though unoptimised queries and an ever growing amount of data are more likely to take the blame for that. High disk I/O in this regard could explain the high system utilisation, however. I found out that I've been bitten by the freebsd-update bug (http://security.freebsd.org/advisories/FreeBSD-EN-07:05.freebsd-update.asc) which replaced my SMP kernel with a GENERIC one and I'm taking corrective action early tomorrow morning, but surely even with just a single CPU the load average should never be as high? Where are those phantom CPU hogging processes? By default top doesn't display system (kernel) processes, which can take up lots of CPU time. To show these, run top with the "-S" flag. -- Bruce ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
