Re: 99% CPU usage in System (Was: Re: vinum in 4.x poor performer?)
On Feb 9, 2005, at 6:34 PM, Marc G. Fournier wrote:
Most odd, there definitely has to be a problem with the Dual-Xeon
ysystem ... doing the same vmstat on my other vinum based system,
running more, but on a Dual-PIII shows major idle time:
# vmstat 5
procs memory pagedisks faults
cpu
r b w avmfre flt re pi po fr sr da0 da1 in sy cs
us sy id
20 1 0 4088636 219556 1664 1 2 1 3058 217 0 0 856 7937 2186
51 15 34
20 1 0 4115372 224220 472 0 0 0 2066 0 0 35 496 2915 745
7 7 86
10 1 0 4125252 221788 916 0 0 0 2513 0 2 71 798 4821 1538
6 11 83
9 1 0 36508 228452 534 0 0 2 2187 0 0 46 554 3384 1027
3 8 89
11 1 0 27672 218828 623 0 6 0 2337 0 0 61 583 2607 679
3 9 88
16 1 05776 220540 989 0 0 0 2393 0 9 32 514 3247 1115
3 8 90
Which leads me further to believe this is a Dual-Xeon problem, and
much further away from believing it has anything to do with software
RAID :(
I only use AMD, so I cannot provide specifics, but look in the BIOS at
boot time and see if there is anything strange looking in the settings.
Chad
On Wed, 9 Feb 2005, Marc G. Fournier wrote:
still getting this:
# vmstat 5
procs memory pagedisks faults
cpu
r b w avmfre flt re pi po fr sr da0 da1 in sy cs
us sy id
11 2 0 3020036 267944 505 2 1 1 680 62 0 0 515 4005 918
7 38 55
19 2 0 3004568 268672 242 0 0 0 277 0 0 3 338 2767 690
1 99 0
21 2 0 2999152 271240 135 0 0 0 306 0 6 9 363 1749 525
1 99 0
13 2 0 3001508 269692 87 0 0 0 24 0 3 3 302 1524 285
1 99 0
17 2 0 3025892 268612 98 0 1 0 66 0 5 6 312 1523 479
3 97 0
Is there a way of determining what is sucking up so much Sys time?
stuff like pperl scripts running and such would use 'user time', no?
I've got some high CPU processes running, but would expect them to be
shooting up the 'user time' ...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME
COMMAND
setiathome 21338 16.3 0.2 7888 7408 ?? RJ9:05PM 0:11.35
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_queuerun -v 0
setiathome 21380 15.1 0.1 2988 2484 ?? RsJ 9:06PM 0:02.42
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
postgresql.org -l pgsql-sql -P10 -p10
setiathome 21384 15.5 0.1 2988 2484 ?? RsJ 9:06PM 0:02.31
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
postgresql.org -l pgsql-docs -P10 -p10
setiathome 21389 15.0 0.1 2720 2216 ?? RsJ 9:06PM 0:02.06
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
postgresql.org -l pgsql-hackers -P10 -p10
setiathome 21386 13.7 0.1 2720 2216 ?? RsJ 9:06PM 0:02.03
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
postgresql.org -l pgsql-ports -P10 -p10
setiathome 21387 13.2 0.1 2724 2220 ?? RsJ 9:06PM 0:01.92
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d
postgresql.org -l pgsql-interfaces -P10 -p10
setiathome 21390 14.6 0.1 2724 2216 ?? RsJ 9:06PM 0:01.93
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -o -d
postgresql.org -l pgsql-performance -P10 -p10
setiathome 21330 12.0 0.2 8492 7852 ?? RJ9:05PM 0:15.55
/usr/bin/perl -wT /dev/fd/3//usr/local/www/mj/mj_wwwusr (perl5.8.5)
setiathome 7864 8.9 0.2 8912 8452 ?? RJ7:20PM 29:54.88
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_trigger -t hourly
Is there some way of finding out where all the Sys Time is being
used? Something more fine grained them what vmstat/top shows?
On Wed, 9 Feb 2005, Loren M. Lang wrote:
On Wed, Feb 09, 2005 at 02:32:30AM -0400, Marc G. Fournier wrote:
Is there a command that I can run that provide me the syscall/sec
value,
that I could use in a script? I know vmstat reports it, but is
there an
easier way the having to parse the output? a perl module maybe, that
already does it?
vmstat shouldn't be too hard to parse, try the following:
vmstat|tail -1|awk '{print $15;}'
To print out the 15th field of vmstat. Now if you want vmstat to
keep
running every five seconds or something, it's a little more
complicated:
vmstat 5|grep -v 'procs\|avm'|awk '{print $15;}'
Thanks ...
On Wed, 9 Feb 2005, Marc G. Fournier wrote:
On Tue, 8 Feb 2005, Dan Nelson wrote:
Details on the array's performance, I think. Software RAID5 will
definitely have poor write performance (logging disks solve that
problem but vinum doesn't do that), but should have excellent read
rates. From this output, however:
systat -v output help:
4 usersLoad 4.64 5.58 5.77
Proc:r p d s wCsw Trp Sys Int Sof Flt
24 9282 949 8414* 678 349 8198
54.6%Sys 0.2%Intr 45.2%User 0.0%Nice 0.0%Idl
Disks da0 da1 da2 da3 da4 pass0 pass1
KB/t 5.32 9.50 12.52 16.00 9.00 0.00 0.00
tps 23 2 4 3 1 0 0
MB/s 0.12 0.01
Re: 99% CPU usage in System (Was: Re: vinum in 4.x poor performer?)
On Wed, 9 Feb 2005, Chad Leigh -- Shire.Net LLC wrote:
On Feb 9, 2005, at 6:34 PM, Marc G. Fournier wrote:
Most odd, there definitely has to be a problem with the Dual-Xeon ysystem
... doing the same vmstat on my other vinum based system, running more, but
on a Dual-PIII shows major idle time:
# vmstat 5
procs memory pagedisks faults cpu
r b w avmfre flt re pi po fr sr da0 da1 in sy cs us sy
id
20 1 0 4088636 219556 1664 1 2 1 3058 217 0 0 856 7937 2186 51
15 34
20 1 0 4115372 224220 472 0 0 0 2066 0 0 35 496 2915 745 7 7
86
10 1 0 4125252 221788 916 0 0 0 2513 0 2 71 798 4821 1538 6
11 83
9 1 0 36508 228452 534 0 0 2 2187 0 0 46 554 3384 1027 3
8 89
11 1 0 27672 218828 623 0 6 0 2337 0 0 61 583 2607 679 3 9
88
16 1 05776 220540 989 0 0 0 2393 0 9 32 514 3247 1115 3
8 90
Which leads me further to believe this is a Dual-Xeon problem, and much
further away from believing it has anything to do with software RAID :(
I only use AMD, so I cannot provide specifics, but look in the BIOS at boot
time and see if there is anything strange looking in the settings.
Unfortunately, I'm dealing with remote servers, so without something
specific to get a remote tech to check, BIOS related stuff will have to
wait until I can visit the servers persoally :(
Chad
On Wed, 9 Feb 2005, Marc G. Fournier wrote:
still getting this:
# vmstat 5
procs memory pagedisks faults cpu
r b w avmfre flt re pi po fr sr da0 da1 in sy cs us sy
id
11 2 0 3020036 267944 505 2 1 1 680 62 0 0 515 4005 918 7 38
55
19 2 0 3004568 268672 242 0 0 0 277 0 0 3 338 2767 690 1 99
0
21 2 0 2999152 271240 135 0 0 0 306 0 6 9 363 1749 525 1 99
0
13 2 0 3001508 269692 87 0 0 0 24 0 3 3 302 1524 285 1 99
0
17 2 0 3025892 268612 98 0 1 0 66 0 5 6 312 1523 479 3 97
0
Is there a way of determining what is sucking up so much Sys time? stuff
like pperl scripts running and such would use 'user time', no? I've got
some high CPU processes running, but would expect them to be shooting up
the 'user time' ...
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
setiathome 21338 16.3 0.2 7888 7408 ?? RJ9:05PM 0:11.35
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_queuerun -v 0
setiathome 21380 15.1 0.1 2988 2484 ?? RsJ 9:06PM 0:02.42
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org
-l pgsql-sql -P10 -p10
setiathome 21384 15.5 0.1 2988 2484 ?? RsJ 9:06PM 0:02.31
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org
-l pgsql-docs -P10 -p10
setiathome 21389 15.0 0.1 2720 2216 ?? RsJ 9:06PM 0:02.06
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org
-l pgsql-hackers -P10 -p10
setiathome 21386 13.7 0.1 2720 2216 ?? RsJ 9:06PM 0:02.03
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org
-l pgsql-ports -P10 -p10
setiathome 21387 13.2 0.1 2724 2220 ?? RsJ 9:06PM 0:01.92
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -r -d postgresql.org
-l pgsql-interfaces -P10 -p10
setiathome 21390 14.6 0.1 2724 2216 ?? RsJ 9:06PM 0:01.93
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_enqueue -o -d postgresql.org
-l pgsql-performance -P10 -p10
setiathome 21330 12.0 0.2 8492 7852 ?? RJ9:05PM 0:15.55
/usr/bin/perl -wT /dev/fd/3//usr/local/www/mj/mj_wwwusr (perl5.8.5)
setiathome 7864 8.9 0.2 8912 8452 ?? RJ7:20PM 29:54.88
/usr/bin/perl -wT /usr/local/majordomo/bin/mj_trigger -t hourly
Is there some way of finding out where all the Sys Time is being used?
Something more fine grained them what vmstat/top shows?
On Wed, 9 Feb 2005, Loren M. Lang wrote:
On Wed, Feb 09, 2005 at 02:32:30AM -0400, Marc G. Fournier wrote:
Is there a command that I can run that provide me the syscall/sec value,
that I could use in a script? I know vmstat reports it, but is there an
easier way the having to parse the output? a perl module maybe, that
already does it?
vmstat shouldn't be too hard to parse, try the following:
vmstat|tail -1|awk '{print $15;}'
To print out the 15th field of vmstat. Now if you want vmstat to keep
running every five seconds or something, it's a little more complicated:
vmstat 5|grep -v 'procs\|avm'|awk '{print $15;}'
Thanks ...
On Wed, 9 Feb 2005, Marc G. Fournier wrote:
On Tue, 8 Feb 2005, Dan Nelson wrote:
Details on the array's performance, I think. Software RAID5 will
definitely have poor write performance (logging disks solve that
problem but vinum doesn't do that), but should have excellent read
rates. From this output, however:
systat -v output help:
4 usersLoad 4.64 5.58 5.77
Proc:r p d s wCsw Trp Sys Int Sof Flt
24 9282 949
