As mentioned before, IO could give such strange results. I suggest launching
dstat with logging to a file, and analyzing the file afterwards.

Thanks, much appreciated!

This has yielded some interesting data, which I'll attempt to include a few seconds before and after one of these events occurred.

system interrupts per second: Note the ~200x jump to almost 200,000 interrupts per second.
                2907
                6714
                1371
                194218
                2456
                2907

network received: Note the network received ramps up over 5 seconds, peaks at ~50x background, and ramps back down in about 3 seconds. The peak is from the same sample as the 200x sample above.
                108784
                389794
                1070850
                4843956
                352226
                353102
                96392


Everything else looks sane -- there's enough ram, nothing's being swapped out, etc. This is on a private-network server that has a load balancer in front of it, so if it's network related, it wouldn't be misdirected random bits.

Has anyone seen this sort of behavior before? What was the cause? What should I do to figure out how to keep the load averages from flipping out of control?

(This isn't something as lame as a counter rolling over somewhere internal to the kernel, is it? Wouldn't think so, but thought to ask. Running 2.6.18-8.1.8.el5. We could reboot to run 2.6.18-8.1.15 if that'd be a potential fix.)

Thanks for any insight!

best,
Jeff




total cpu usage dsk/total               net/total               system  
usr     sys     read    writ    recv    send    int     csw
10.5    3.25    0       409600  108784  72286   2907    20376
3.99    2.993   0       319488  389794  661170  6714    23941
0.25    0.25    0       720896  1070850 1189720 1371    16648
9.167   90.442  12288   1122304 4843956 399998  194218  55433
56.931  16.832  0       1273856 352226  334506  2456    12844
46.25   20      0       454656  353102  384496  2907    20631
24.25   1.25    0       3260416 96392   72316   1342    17307
23.25   2.25    0       610304  91086   71194   1458    17584
10.973  1.496   0       0       84192   46276   1349    18135
0       0       0       94208   71892   33304   1220    16979
0.25    0.25    0       126976  71184   47576   1268    16973
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Reply via email to