Re: [rt-users] apache 100% cpu usage
I tried to examine your idea about big search. I did some of them (i.e. 128000 results) but httpd process only peaked (still not 100%) and in a few seconds get back to the normal state. It seems search with large result could not be the exact source if this problem Best Regards From: rt-users-boun...@lists.bestpractical.com [mailto:rt-users-boun...@lists.bestpractical.com] On Behalf Of Payam Poursaied Sent: Sunday, March 20, 2011 8:55 PM To: 'Ruslan Zakirov' Cc: rt-users@lists.bestpractical.com Subject: Re: [rt-users] apache 100% cpu usage Hi Ruslan Thank you for your response. There isn’t any end! Even after a week! All modules are updated to the latest available port in FreeBSD You gave me a good point to check, but still I am curious to know how such problems could be debugged. Do you have any idea on this issue ? Best Regards From: Ruslan Zakirov [mailto:ruslan.zaki...@gmail.com] Sent: Sunday, March 20, 2011 7:39 PM To: Payam Poursaied Cc: rt-users@lists.bestpractical.com Subject: Re: [rt-users] apache 100% cpu usage I see storable perl module there, so it's sessions handling. Look at updates for the module. Try to wait to see if it ends. If it ends then may be It's result of a big search that stored in the session. Newer versions of RT don't store big search results in sessions, only limited amount of records. Check sessions table for big records. Regards, Ruslan. From phone. 20.03.2011 11:19 пользователь Payam Poursaied m...@payam124.com написал: Hi all We are using rt3.8.8 and we faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd 95331 www 1 46 0 205M 83464K sbwait 12 0:01 1.66% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. But I could not go further more. I have also consulted with apache-freebsd mailing list but there wasn't any success. Could anyone help me to drill down this problem? Best Regards Payam Poursaied Using next says: (gdb) next Cannot find bounds of current function And here below is output of where command in gdb66. (gdb) where #0 0x00080093a73b in ?? () from /lib/libc.so.7 #1 0x00080093d505 in ?? () from /lib/libc.so.7 #2 0x0008009401bc in ?? () from /lib/libc.so.7 #3 0x000800942ec4 in malloc () from /lib/libc.so.7 #4 0x000803079946 in Perl_safesysmalloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #5 0x00080308cee0 in Perl_av_extend () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #6 0x00080308dabe in Perl_av_store () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #7 0x00080750de53 in retrieve_byte () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #8 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #9 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #10 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #11 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #12 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #13 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #14 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #15 0x00080751298c in retrieve_blessed () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #16 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #17 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #18 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #19 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #20 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1
Re: [rt-users] apache 100% cpu usage
I found out a while ago that looping emails would cause RT to go crazy like that. I disabled $LoopsToRTOwner and the problem disappeared. I didn't have time to debug the issue further... -- Mathieu Longtin 1-514-803-8977 On Sun, Mar 20, 2011 at 4:19 AM, Payam Poursaied m...@payam124.com wrote: Hi all We are using rt3.8.8 and we faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 77051 www 1 1180 208M 86080K CPU44 87.6H 100.00% httpd 9403 www 1 1180 215M 93340K CPU22 65.0H 100.00% httpd 78430 www 1 1180 226M 100M CPU00 58:57 100.00% httpd 95332 www 1 500 205M 83624K sbwait 12 0:02 4.49% httpd 89664 www 1 460 210M 88144K sbwait 11 0:05 2.59% httpd 95331 www 1 460 205M 83464K sbwait 12 0:01 1.66% httpd 46157 www 1 470 258M 134M sbwait 15 3:03 1.17% httpd 89493 www 1 440 212M 90152K accept 13 0:20 0.10% httpd 22801 www 1 440 264M 137M accept 13 5:40 0.00% httpd 25440 www 1 440 275M 145M accept 11 5:35 0.00% httpd 30396 www 1 440 262M 135M accept 14 4:47 0.00% httpd 82456 www 1 440 233M 109M accept 12 1:37 0.00% httpd 80407 www 1 440 239M 110M accept 10 0:49 0.00% httpd 80097 www 1 440 265M 135M accept 1 0:35 0.00% httpd 88622 www 1 460 216M 94588K accept 15 0:16 0.00% httpd 88968 www 1 440 208M 86316K accept 6 0:03 0.00% httpd 89686 www 1 440 201M 80216K accept 11 0:01 0.00% httpd 95330 www 1 440 201M 77088K sbwait 10 0:00 0.00% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. But I could not go further more. I have also consulted with apache-freebsd mailing list but there wasn't any success. Could anyone help me to drill down this problem? Best Regards Payam Poursaied Using next says: (gdb) next Cannot find bounds of current function And here below is output of where command in gdb66. (gdb) where #0 0x00080093a73b in ?? () from /lib/libc.so.7 #1 0x00080093d505 in ?? () from /lib/libc.so.7 #2 0x0008009401bc in ?? () from /lib/libc.so.7 #3 0x000800942ec4 in malloc () from /lib/libc.so.7 #4 0x000803079946 in Perl_safesysmalloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #5 0x00080308cee0 in Perl_av_extend () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #6 0x00080308dabe in Perl_av_store () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #7 0x00080750de53 in retrieve_byte () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #8 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #9 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #10 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #11 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #12 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #13 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #14 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #15 0x00080751298c in retrieve_blessed () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #16 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #17 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #18 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #19 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #20 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #21 0x00080750d55f in do_retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #22 0x00080750dd28 in XS_Storable_mretrieve () from
Re: [rt-users] apache 100% cpu usage
I see storable perl module there, so it's sessions handling. Look at updates for the module. Try to wait to see if it ends. If it ends then may be It's result of a big search that stored in the session. Newer versions of RT don't store big search results in sessions, only limited amount of records. Check sessions table for big records. Regards, Ruslan. From phone. 20.03.2011 11:19 пользователь Payam Poursaied m...@payam124.com написал: Hi all We are using rt3.8.8 and we faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd 95331 www 1 46 0 205M 83464K sbwait 12 0:01 1.66% httpd 46157 www 1 47 0 258M 134M sbwait 15 3:03 1.17% httpd 89493 www 1 44 0 212M 90152K accept 13 0:20 0.10% httpd 22801 www 1 44 0 264M 137M accept 13 5:40 0.00% httpd 25440 www 1 44 0 275M 145M accept 11 5:35 0.00% httpd 30396 www 1 44 0 262M 135M accept 14 4:47 0.00% httpd 82456 www 1 44 0 233M 109M accept 12 1:37 0.00% httpd 80407 www 1 44 0 239M 110M accept 10 0:49 0.00% httpd 80097 www 1 44 0 265M 135M accept 1 0:35 0.00% httpd 88622 www 1 46 0 216M 94588K accept 15 0:16 0.00% httpd 88968 www 1 44 0 208M 86316K accept 6 0:03 0.00% httpd 89686 www 1 44 0 201M 80216K accept 11 0:01 0.00% httpd 95330 www 1 44 0 201M 77088K sbwait 10 0:00 0.00% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. But I could not go further more. I have also consulted with apache-freebsd mailing list but there wasn't any success. Could anyone help me to drill down this problem? Best Regards Payam Poursaied Using next says: (gdb) next Cannot find bounds of current function And here below is output of where command in gdb66. (gdb) where #0 0x00080093a73b in ?? () from /lib/libc.so.7 #1 0x00080093d505 in ?? () from /lib/libc.so.7 #2 0x0008009401bc in ?? () from /lib/libc.so.7 #3 0x000800942ec4 in malloc () from /lib/libc.so.7 #4 0x000803079946 in Perl_safesysmalloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #5 0x00080308cee0 in Perl_av_extend () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #6 0x00080308dabe in Perl_av_store () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #7 0x00080750de53 in retrieve_byte () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #8 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #9 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #10 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #11 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #12 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #13 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #14 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #15 0x00080751298c in retrieve_blessed () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #16 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #17 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #18 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #19 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #20 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #21 0x00080750d55f in do_retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #22 0x00080750dd28 in XS_Storable_mretrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #23 0x000803090620 in Perl_pp_entersub () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #24 0x00080308eb9e in Perl_runops_standard () from
Re: [rt-users] apache 100% cpu usage
Hi Do you mean that looping email would be the source of the problem? I haven’t seen any looped email in our system. Also I have defined all email address in $RTAddressRegexp. So I think it would prevent loop. Isn’t it? From: Mathieu Longtin [mailto:math...@closetwork.org] Sent: Sunday, March 20, 2011 4:02 PM To: Payam Poursaied Cc: rt-users@lists.bestpractical.com Subject: Re: [rt-users] apache 100% cpu usage I found out a while ago that looping emails would cause RT to go crazy like that. I disabled $LoopsToRTOwner and the problem disappeared. I didn't have time to debug the issue further... -- Mathieu Longtin 1-514-803-8977 On Sun, Mar 20, 2011 at 4:19 AM, Payam Poursaied m...@payam124.com wrote: Hi all We are using rt3.8.8 and we faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 77051 www 1 1180 208M 86080K CPU44 87.6H 100.00% httpd 9403 www 1 1180 215M 93340K CPU22 65.0H 100.00% httpd 78430 www 1 1180 226M 100M CPU00 58:57 100.00% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. But I could not go further more. I have also consulted with apache-freebsd mailing list but there wasn't any success. Could anyone help me to drill down this problem? Best Regards Payam Poursaied Using next says: (gdb) next Cannot find bounds of current function And here below is output of where command in gdb66. (gdb) where #0 0x00080093a73b in ?? () from /lib/libc.so.7 #1 0x00080093d505 in ?? () from /lib/libc.so.7 #2 0x0008009401bc in ?? () from /lib/libc.so.7 #3 0x000800942ec4 in malloc () from /lib/libc.so.7 #4 0x000803079946 in Perl_safesysmalloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so smime.p7s Description: S/MIME cryptographic signature
Re: [rt-users] apache 100% cpu usage
Hi Ruslan Thank you for your response. There isn’t any end! Even after a week! All modules are updated to the latest available port in FreeBSD You gave me a good point to check, but still I am curious to know how such problems could be debugged. Do you have any idea on this issue ? Best Regards From: Ruslan Zakirov [mailto:ruslan.zaki...@gmail.com] Sent: Sunday, March 20, 2011 7:39 PM To: Payam Poursaied Cc: rt-users@lists.bestpractical.com Subject: Re: [rt-users] apache 100% cpu usage I see storable perl module there, so it's sessions handling. Look at updates for the module. Try to wait to see if it ends. If it ends then may be It's result of a big search that stored in the session. Newer versions of RT don't store big search results in sessions, only limited amount of records. Check sessions table for big records. Regards, Ruslan. From phone. 20.03.2011 11:19 пользователь Payam Poursaied m...@payam124.com написал: Hi all We are using rt3.8.8 and we faced with a problem using apache 1.3.42 on FreeBSD 8.0-RELEASE. The problem is that httpd processes frequently reach 100% CPU usage and stay at this level of CPU usage. This is a sample of top output. PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 77051 www 1 118 0 208M 86080K CPU4 4 87.6H 100.00% httpd 9403 www 1 118 0 215M 93340K CPU2 2 65.0H 100.00% httpd 78430 www 1 118 0 226M 100M CPU0 0 58:57 100.00% httpd 95332 www 1 50 0 205M 83624K sbwait 12 0:02 4.49% httpd 89664 www 1 46 0 210M 88144K sbwait 11 0:05 2.59% httpd 95331 www 1 46 0 205M 83464K sbwait 12 0:01 1.66% httpd I think there might be a bug that caused something like endless loop but I am newbie to gdb and debugging and I could not find any relevant point to get into it. I tried to compile apache in debug mode and then wait to see a suspicious process and then attached GDB66 to that cpu intensive httpd process to find out what is going on. But I could not go further more. I have also consulted with apache-freebsd mailing list but there wasn't any success. Could anyone help me to drill down this problem? Best Regards Payam Poursaied Using next says: (gdb) next Cannot find bounds of current function And here below is output of where command in gdb66. (gdb) where #0 0x00080093a73b in ?? () from /lib/libc.so.7 #1 0x00080093d505 in ?? () from /lib/libc.so.7 #2 0x0008009401bc in ?? () from /lib/libc.so.7 #3 0x000800942ec4 in malloc () from /lib/libc.so.7 #4 0x000803079946 in Perl_safesysmalloc () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #5 0x00080308cee0 in Perl_av_extend () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #6 0x00080308dabe in Perl_av_store () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #7 0x00080750de53 in retrieve_byte () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #8 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #9 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #10 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #11 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #12 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #13 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #14 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #15 0x00080751298c in retrieve_blessed () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #16 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #17 0x00080750ce29 in retrieve_ref () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #18 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #19 0x0008075113b8 in retrieve_hash () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #20 0x00080750c66b in retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #21 0x00080750d55f in do_retrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #22 0x00080750dd28 in XS_Storable_mretrieve () from /usr/local/lib/perl5/site_perl/5.10.1/mach/auto/Storable/Storable.so #23 0x000803090620 in Perl_pp_entersub () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #24 0x00080308eb9e in Perl_runops_standard () from /usr/local/lib/perl5/5.10.1/mach/CORE/libperl.so #25 0x0008030c1ee3 in S_docatch () from /usr/local/lib/perl5/5.10.1/mach
