Re: [Catalyst] Intermittent blank pages
On Thu, Aug 28, 2008 at 05:44:31PM +0200, Tobias Kremer wrote: Quoting [EMAIL PROTECTED]: Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 09:02:24 AM: Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( Are they delivered instantly or after a long pause? Instantly. I just noticed that shortly after a series of those errors appear in the error log, the process gets killed off (and restarted) by daemontools' softlimit due to memory constraints. You might want to load Catalyst::Plugin::AutoRestart as well so the processes restart themselves gracefully at the end of a request before it hits the softlimit. I notice downthread you've solved the underlying problem, but adding AutoRestart into your setup would mean that if something similar goes wrong again you'll avoid the firing blanks problem. -- Matt S Trout Need help with your Catalyst or DBIx::Class project? Technical Directorhttp://www.shadowcat.co.uk/catalyst/ Shadowcat Systems Ltd. Want a managed development or deployment platform? http://chainsawblues.vox.com/http://www.shadowcat.co.uk/servers/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
On Wed, Sep 03, 2008 at 01:45:47PM +0300, Octavian Rasnita wrote: From: Matt S Trout [EMAIL PROTECTED] You might want to load Catalyst::Plugin::AutoRestart as well so the processes restart themselves gracefully at the end of a request before it hits the softlimit. Do you know if there is a similar solution that can be used under Windows? Write one and send it to the authors, I'm sure they'd be happy to take a patch. -- Matt S Trout Need help with your Catalyst or DBIx::Class project? Technical Directorhttp://www.shadowcat.co.uk/catalyst/ Shadowcat Systems Ltd. Want a managed development or deployment platform? http://chainsawblues.vox.com/http://www.shadowcat.co.uk/servers/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages - possibly solved
Quoting Tomas Doran [EMAIL PROTECTED]: On 28 Aug 2008, at 19:32, [EMAIL PROTECTED] wrote: That was my guess, too. But if the problem is really situated there it should have occurred much earlier. Nothing dramatically changed in our setup or application. I'll try a rollback of the Rose::HTML::Form module to eliminate it from the equation. Any other thoughts are still greatly appreciated! Are you leaking memory (or is Rose leaking memory for you)? Using something like Devel::Leak::Object could help you here.. Okay guys, Rose::HTML::Form seems to leak memory like mad. I've tried every single version from the most current one (0.554) downwards and checked the memory usage with a simple Catalyst application. The process size became stable after downgrading to Rose::HTML::Object 0.549. So the latest six versions have a huge memory leak (the cat process grew up to 100 MB with nothing but a simple form after using ab to hit the app with 1000 requests)! I'll file a bug report and take a look at what's going on under the hood. --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Intermittent blank pages
Two days ago I suddenly started receiving intermittent blank pages. Immediately reloading the page led to the expected response. Looking at the error logs of the past two days I can see loads of the following error: Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/site_perl/5.8.8/Catalyst/Engine/FastCGI.pm line 169. It appears that there just isn't any body data to process. This error was absolutely non-existent until a few days ago. AFAICT the only things that have changed since then is the session table now being InnoDB instead of MyISAM and Rose::HTML::Form has been updated to the latest version (5.7014). Due to there being no additional error messages or warnings I don't know how to debug this. Does anybody here have an idea what could be going wrong? I'm using Catalyst 5.7010. Thanks! --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Tobias Kremer wrote: Two days ago I suddenly started receiving intermittent blank pages. Immediately reloading the page led to the expected response. Looking at the error logs of the past two days I can see loads of the following error: Use of uninitialized value in concatenation (.) or string at /usr/local/lib/perl5/site_perl/5.8.8/Catalyst/Engine/FastCGI.pm line 169. It appears that there just isn't any body data to process. This error was absolutely non-existent until a few days ago. AFAICT the only things that have changed since then is the session table now being InnoDB instead of MyISAM and Rose::HTML::Form has been updated to the latest version (5.7014). Due to there being no additional error messages or warnings I don't know how to debug this. Does anybody here have an idea what could be going wrong? I'm using Catalyst 5.7010. Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... Cheers, Sebastian ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 09:02:24 AM: Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( --Tobias Are they delivered instantly or after a long pause? ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
On Thu, Aug 28, 2008 at 7:02 AM, Tobias Kremer [EMAIL PROTECTED] wrote:
Quoting Sebastian Willert [EMAIL PROTECTED]:
Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver
blank pages if the back-end times out. Unfortunately I haven't found
a way to avoid this behavior, but maybe someone on the list can help
out ...
No, we're using lighttpd and nothing changed on that front during the last two
days :(
--Tobias
Are you using an external fastcgi socket?
You may want to isolate if it is a problem in the Catalyst or
somewhere upstream by adding something into the dispatcher (probably
after -finalize) or you can add sub use_stats { 1 } to MyApp.pm,
which will then log the stat info to $c-log-info (so make sure you
have that enabled)
That will help shed some light on where things are going blank.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Quoting [EMAIL PROTECTED]: Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 09:02:24 AM: Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( Are they delivered instantly or after a long pause? Instantly. I just noticed that shortly after a series of those errors appear in the error log, the process gets killed off (and restarted) by daemontools' softlimit due to memory constraints. Afterwards it takes some time until the error reappears. But we have this exact setup running for over a year now and never had said problem. --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Tobias Kremer wrote: Quoting [EMAIL PROTECTED]: Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 09:02:24 AM: Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( Are they delivered instantly or after a long pause? Instantly. I just noticed that shortly after a series of those errors appear in the error log, the process gets killed off (and restarted) by daemontools' softlimit due to memory constraints. Afterwards it takes some time until the error reappears. But we have this exact setup running for over a year now and never had said problem. A chunk of RAM going bad on the system? I've seen similar problems, though they were not running Cat or lighttpd. Has the usage of the application increased or the size of the data being entered/returned/processed? Rod -- --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 10:44:31 AM: Quoting [EMAIL PROTECTED]: Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 09:02:24 AM: Quoting Sebastian Willert [EMAIL PROTECTED]: Are you using Perlbal in front of FastCGI? Perlbal is prone to deliver blank pages if the back-end times out. Unfortunately I haven't found a way to avoid this behavior, but maybe someone on the list can help out ... No, we're using lighttpd and nothing changed on that front during the last two days :( Are they delivered instantly or after a long pause? Instantly. I just noticed that shortly after a series of those errors appear inthe error log, the process gets killed off (and restarted) by daemontools' softlimit due to memory constraints. Afterwards it takes some time until the errorreappears. But we have this exact setup running for over a year now and never had said problem. --Tobias Is it possible that the concat error and resstart are related (memory load) but the blank pages are due to the restart killing (restarting) the procs off before the fastcgi payload is sent? -Wade ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
On 28.08.2008, at 18:48, [EMAIL PROTECTED] wrote: Is it possible that the concat error and resstart are related (memory load) but the blank pages are due to the restart killing (restarting) the procs off before the fastcgi payload is sent That was my guess, too. But if the problem is really situated there it should have occurred much earlier. Nothing dramatically changed in our setup or application. I'll try a rollback of the Rose::HTML::Form module to eliminate it from the equation. Any other thoughts are still greatly appreciated! --Tobias ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
Tobias Kremer [EMAIL PROTECTED] wrote on 08/28/2008 12:29:13 PM: On 28.08.2008, at 18:48, [EMAIL PROTECTED] wrote: Is it possible that the concat error and resstart are related (memory load) but the blank pages are due to the restart killing (restarting) the procs off before the fastcgi payload is sent That was my guess, too. But if the problem is really situated there it should have occurred much earlier. Nothing dramatically changed in our setup or application. I'll try a rollback of the Rose::HTML::Form module to eliminate it from the equation. Any other thoughts are still greatly appreciated! --Tobias Change happens over time too, is the dataset that is being loaded/used getting larger over time and now hitting a tipping point? Maybe the new Rose modules are tunes more speedspace? How often did the procs get restarted due to memory hard limits in the past? Is that behavior in action new as well? -Wade ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Intermittent blank pages
On 28 Aug 2008, at 19:32, [EMAIL PROTECTED] wrote: That was my guess, too. But if the problem is really situated there it should have occurred much earlier. Nothing dramatically changed in our setup or application. I'll try a rollback of the Rose::HTML::Form module to eliminate it from the equation. Any other thoughts are still greatly appreciated! Are you leaking memory (or is Rose leaking memory for you)? Using something like Devel::Leak::Object could help you here.. Assuming your issue doesn't fall that easily, you need to instrument the app in some way, and analyze it's behavior. Questions you should be working out how to answer: . Is the memory useage is going up gradually, or in steps? . How often are processes getting killed? . Is the memory useage associated with some of your app's URLs / queries, but not others? Why not try doing something like printing the requested URL, any other relevant info, and the total memory size of the process, and amount it climbed by this hit to the error log for each hit? (A closure to hold how much RAM you were using last time, and cribbing the source of Apache2::SizeLimit to work out how it works it's magic are your friends here). Cheers t0m ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
