Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Hi, since initially it seems like the high load may be due to I/O wait Maybe this will help you to identify the IO loading process: http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5 Thanks for the suggestion, I did install dstat earlier while trying to figure things out

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Hi, You should also try out atop instead of just using top. The major advantage is that it gives you more information about the disk and network utilization. Thanks for the tip, I tried it and if the red lines are any indication, it seems that atop thinks my disks (md raid 1) are the

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Hi, Dstat could at least tell you if your problem is CPU or I/O. This was the result of running the following command which I obtained from reading up about two weeks ago when I started trying to investigate the abnormal server behaviour. dstat -c --top-cpu -d --top-bio --top-latency usr sys

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Chan Chung Hang Christopher
Look at the first two columns. What column have higher numbers? If r, you're CPU-bound. If b, you're I/O bound. procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 8 1

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Christoph Maser
Am Donnerstag, den 31.12.2009, 12:34 +0100 schrieb Chan Chung Hang Christopher: Look at the first two columns. What column have higher numbers? If r, you're CPU-bound. If b, you're I/O bound. procs ---memory-- ---swap-- -io --system-- -cpu-- r b

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Chan Chung Hang Christopher
Christoph Maser wrote: Am Donnerstag, den 31.12.2009, 12:34 +0100 schrieb Chan Chung Hang Christopher: Look at the first two columns. What column have higher numbers? If r, you're CPU-bound. If b, you're I/O bound. procs ---memory-- ---swap-- -io --system--

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Hi, Yes, these figures indicate that you are fairly close to being cpu bound. What kind of filtering are you doing? If you have any connection tracking/state related rules set, you will need to be using a fair amount of cpu. Initially, when the load start going up, I had thought the APF

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Chan Chung Hang Christopher
Noob Centos Admin wrote: Hi, Yes, these figures indicate that you are fairly close to being cpu bound. What kind of filtering are you doing? If you have any connection tracking/state related rules set, you will need to be using a fair amount of cpu. Initially, when the load start going

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Hi, I do not know about now but I had to unload the modules in question. Just clearing the rules was not enough to ensure that the netfilter connection tracking modules were not using any cpu at all. Thanks for pointing this out. Being a noob admin as my pseudonym states, I'd assumed stopping

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
I initiated services shutdown as previously planned and once the external services like exim, dovecot, httpd, crond (because it kept restarting these services), the problem child stood out like a sore thumb. There was two exim instances that didn't go away despite service exim stop. Once I killed

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Noob Centos Admin
Just an concluding update to anybody who might be interested :) My apologies for blaming spamassassin in the earlier email. It was taking so long because of the real problem. Apparently the odd exim processes that was related to the mail loop problem I nipped was still the culprit. I had

Re: [CentOS] Find reason for heavy load

2009-12-31 Thread Ugo Bellavance
On 2009-12-31 15:13, Noob Centos Admin wrote: Just an concluding update to anybody who might be interested :) My apologies for blaming spamassassin in the earlier email. It was taking so long because of the real problem. Apparently the odd exim processes that was related to the mail loop

Re: [CentOS] Find reason for heavy load

2009-12-30 Thread Thomas Harold
On 12/29/2009 11:44 PM, Noob Centos Admin wrote: My Centos 5 server has seen the average load jumped through the roof recently despite having no major additional clients placed on it. Previously, I was looking at an average of less than 0.6 load, I had a monitoring script that sends an email

Re: [CentOS] Find reason for heavy load

2009-12-30 Thread Ugo Bellavance
On 2009-12-29 23:44, Noob Centos Admin wrote: My Centos 5 server has seen the average load jumped through the roof recently despite having no major additional clients placed on it. Previously, I was looking at an average of less than 0.6 load, I had a monitoring script that sends an email

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread John R Pierce
Noob Centos Admin wrote: My Centos 5 server has seen the average load jumped through the roof recently despite having no major additional clients placed on it. Previously, I was looking at an average of less than 0.6 load, I had a monitoring script that sends an email warning me if the

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread Ross Walker
On Dec 29, 2009, at 11:44 PM, Noob Centos Admin centos.ad...@gmail.com wrote: My Centos 5 server has seen the average load jumped through the roof recently despite having no major additional clients placed on it. Previously, I was looking at an average of less than 0.6 load, I had a

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread Noob Centos Admin
Hi, last time I saw something like that, it was a bunch of chinese 'bots' hammering on my public services like ssh. another admin had turned pop3 on too, this created a very heavy load yet they didn't show up in top (bunches of pop3 and ssh processes showed up in ps -auxww, however, plug

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread Noob Centos Admin
Hi, Try blocking the IPs on the router and see if that helps. Unfortunately the server's in a DC so the router is not under our control. You can also run iostat and look at the disk usage which also generates load. I did try iostat and its iowait% did coincide with top's report, which is

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread John R Pierce
Noob Centos Admin wrote: However, iostat reports much lower %user and $system compared to top running at the same time so I'm not quite sure if I can rely on its figures. ... iostat Linux 2.6.18-128.1.16.el5xen 12/30/2009 avg-cpu: %user %nice %system %iowait %steal %idle

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread Ross Walker
On Dec 30, 2009, at 1:05 AM, Noob Centos Admin centos.ad...@gmail.com wrote: Hi, Try blocking the IPs on the router and see if that helps. Unfortunately the server's in a DC so the router is not under our control. That sucks, oh well. You can also run iostat and look at the disk

Re: [CentOS] Find reason for heavy load

2009-12-29 Thread Christoph Maser
Am Mittwoch, den 30.12.2009, 05:44 +0100 schrieb Noob Centos Admin: since initially it seems like the high load may be due to I/O wait Maybe this will help you to identify the IO loading process: http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5 Chris financial.com AG