I write a driver for lcdproc that recreates some procps | top | ps
functionality by reading /proc/[number]/stat.
The driver uses pthreads, with a varying number of threads depending on
tasks.
Updating my driver from 32 bit etch to 64 bit lenny I've found similar
'errors' in reported % cpu values.
Under etch, top reports on LCDd ( the main binary of the lcdrpoc
package) like this
top -p 3127; 'H' for thread view; 'I' Irix mode off :
top - 13:32:27 up 23 min, 2 users, load average: 0.06, 0.04, 0.00
Tasks: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.1%sy, 0.0%ni, 99.5%id, 0.1%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 904360k total, 368916k used, 535444k free, 15420k buffers
Swap: 3068372k total, 0k used, 3068372k free, 171808k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3413 nobody 15 0 36196 1408 1008 S 0 0.2 0:00.77 LCDd
3127 nobody 15 0 36196 1408 1008 S 0 0.2 0:00.00 LCDd
3131 nobody 15 0 36196 1408 1008 S 0 0.2 0:00.00 LCDd
3132 nobody 15 0 36196 1408 1008 S 0 0.2 0:00.00 LCDd
3412 nobody 15 0 36196 1408 1008 S 0 0.2 0:00.00 LCDd
Under etch my lcdproc driver is almost always at 0% cpu.
On Lenny it can look like this:
top - 10:13:03 up 1:46, 6 users, load average: 0.05, 0.04, 0.00
Tasks: 7 total, 0 running, 7 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.4%us, 0.3%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 2063316k total, 436268k used, 1627048k free, 632k buffers
Swap: 3068372k total, 0k used, 3068372k free, 210716k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4629 nobody 20 0 56324 1524 1092 S 99.9 0.1 0:00.86 LCDd
4632 nobody 20 0 56324 1524 1092 S 8.2 0.1 0:00.26 LCDd
4631 nobody 20 0 56324 1524 1092 S 0.0 0.1 0:00.00 LCDd
4633 nobody 20 0 56324 1524 1092 S 0.0 0.1 0:00.00 LCDd
This is similar to the irreconcilable difference Igno Saitz's noted
between the high idle % an the per process and per thread % cpu numbers.
The LCDd driver I am monitoring in top reads /proc/[number]/stat in a
manner similar to the procps utilities and writes it to a log with other
performance per watt and thermal management data. ( excuse the long
.csv line mess in this short sample ..)
debian:/mnt/var/log# grep uptime CF.csv -A 2 | tail -n 3
Sun Apr 12 12:11:48
2009,uptime,load,top1,%,top2,%,top3,%,top4,%,WoW%,LCDd%,Xorg%,gnome-%,w83627hf-isa-0290:temp1,w8:temp2,k8temp:Core0
Temp,k8temp:Core1 Temp,/dev/sda:SAMSUNG
HD501LJ,WattNode:watts,inflow:ml/s,^radfan:pwr,psu-gpu1,psu-gpu2,psu-gpu3,...warm1,...warm2,...warm3,hdd-psu1,hdd-psu2,gpu-cpu1,gpu-cpu2,gpu-cpu3,pfl-hdd1,pfl-hdd2,pfl-hdd3,cool1...,cool2...,cool3...,1start,2,3,4,5,6,7,8end,cpu-mbb1,cpu-mbb2,cpu-mbb3,watts
1 ,2 ,3 ,4 , ,6 , ,8 , ,10 , ,12
,13 ,14 ,15 ,16 ,17 ,18
,19 ,20 ,21 ,22 ,23
,24 ,25 ,26 ,27 ,28 ,29 ,30 ,31
,32 ,33 ,34 ,35 ,36 ,37 ,38 ,39
,40 ,41 , , , , , , ,48 ,49 ,50 ,51 ,52
Sun Apr 12 12:11:51
2009,285,3.4,LCDd,4.84,Xorg,2.99,rsyslogd,2.56,gnome-terminal,0.71,0.00,4.84,2.99,1.42,36,34,34,32,28.000000,119.0090,12.1414,000,29.9375,30.1250,30.1875,31.0625,31.0000,31.0000,85.0000,29.5625,30.8125,30.7500,85.0000,29.9375,29.9375,30.0625,29.6250,29.8125,30.0625,26.5000,26.3750,25.8750,25.6875,29.8750,25.3750,25.9375,23.7500,31.1250,31.0625,31.3125,0.00
To cull less pertinant data:
grep "Fri Apr 10 18:[012]" CF.csv | cut -d, -f 1,3,13,21
Fri Apr 10 18:08:50 2009,3.8,2.52,123.1951
Fri Apr 10 18:08:47 2009, load, LCDd%, WattNode:watts
Fri Apr 10 18:08:50 2009, 3.8, 2.52, 123.1951
Fri Apr 10 18:09:50 2009, 3.3, 3.03, 120.5024
Fri Apr 10 18:10:50 2009, 2.4, 5.33, 126.1047
Fri Apr 10 18:11:50 2009, 0.6, 4.00, 125.7203
Fri Apr 10 18:12:50 2009, 1.6, 4.77, 119.7478
Fri Apr 10 18:13:50 2009, 0.6, 3.86, 124.0100
Fri Apr 10 18:14:51 2009, 0.5, 4.22, 120.8644
Fri Apr 10 18:15:51 2009, 0.5, 5.50, 122.5348
Fri Apr 10 18:16:51 2009, 0.4, 1.29, 117.5087
Fri Apr 10 18:17:52 2009, 0.7, 0.62, 123.3697
Fri Apr 10 18:18:52 2009, 0.8, 0.66, 121.0897
Fri Apr 10 18:19:52 2009, 0.3, 1.00, 116.8656
Fri Apr 10 18:20:52 2009, 0.3, 0.50, 121.0964
Fri Apr 10 18:21:52 2009, 0.2, 0.52, 123.4716
Fri Apr 10 18:22:53 2009, 0.3, 0.29, 120.5666
Fri Apr 10 18:23:53 2009, 0.3, 0.39, 118.0608
Fri Apr 10 18:24:53 2009, 0.3, 1.95, 121.7365
Fri Apr 10 18:25:53 2009, 0.2, 0.50, 121.3721
Fri Apr 10 18:26:53 2009, 0.2, 0.28, 122.9431
Fri Apr 10 18:27:54 2009, 0.3, 0.52, 120.4294
Fri Apr 10 18:28:54 2009, 0.3, 0.44, 105.9310
Fri Apr 10 18:29:54 2009, 0.4, 0.93, 124.5159
On lenny (above) , load + LCDd% is often > 100, while load + idle is
neatly 100.
I included the column WattNode:watts in the above. It is a digital
watt-meter measuring how much electricity the computer is using ( minus
monitor) out of the wall socket. If I put a large sectien of the .csv
log into mysql and ran some statistics on it, or put it into oppenoffice
calc and made some charts, I am confident that it would show that the
watt usage correlates highly with total CPU load and not with the
anomolously high LCDd numbers. This is a bit of a tangential, indirect,
or off the beaten path argument for top's cpu % numbers being off -- but
I have it and watts out of the wall socket <> cpu % does not lie.
The lcdproc /proc/[number]/stat readings are synchronously erroneous
with the top and ps numbers, I think.
I think this bug is not properly in the procps package.
I propose it comes from the lenny 2.6.26 kernel kernel writing erroneous
values to /proc/[number]/stat for processes that use pthreads.
Checking 'man proc' on my 2.6.18 kernel etch and 2.6.26 lenny installs I
see that there were changes to the structure and content of the
/proc/[number]/stat fields. I suspect some pthreads specific error in
the __kernel__ code that writes /proc/[number]/stat.
I think it affects both 32 bit and 64 bit kernels, both distribution
kernels and my custom kernels, and that magnitude and consistency of the
errors vary with kernel .config ( processor type and features,
scheduler? cpu-freq scaling ? powernow ? virtialization ? guest os? )
options. I do not see a clear pattern to the error.
Stewart Putnam
stewartputnam at comcast dot net
--
To UNSUBSCRIBE, email to [email protected]
with a subject of "unsubscribe". Trouble? Contact [email protected]
