No errors in /var/log/messages (Amazon Linux)
No file system more than 70% full
Plenty of RAM
CPU not being taxed at all
112 running processes
sudo lsof | wc -l
11590
Nothing obviously askew
Thanks Pascal

On 27 Feb 2014, at 14:36, Pascal Robert <prob...@macti.ca> wrote:

> You might be out of system resources (file open limit, etc.). Any errors in 
> the system log (/var/log/messages in RedHat/CentOS/Amazon Linux)?
> 
> De: "John Pollard" <j...@pollardweb.com>
> À: "Webobjects-deploy@lists.apple.com" <webobjects-deploy@lists.apple.com>
> Envoyé: Jeudi 27 Février 2014 09:34:19
> Objet: Re: Intermittent The requested application was not found on this       
>  server.
> 
> I am getting more intermittent app not found on server.
> Looking at the WO apache module log:
> 
> A good request sequence logged looks like this:
> Info: received ->200 Apple
> Debug: <WebObjects Apache Module> new translate: 
> /ires/8206/GOHWeb/miscRes/gohStyle11.css
> Debug: <WebObjects Apache Module> translate - DECLINED: 
> /ires/8206/GOHWeb/miscRes/gohStyle11.css
> Debug: WebObjects_handler declined! /ires/8206/GOHWeb/miscRes/gohStyle11.css
> Debug: <WebObjects Apache Module> new translate: /favicon.ico
> Debug: <WebObjects Apache Module> translate - DECLINED: /favicon.ico
> Debug: WebObjects_handler declined! /favicon.ico 
> Info: New request is GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html 
> HTTP/1.1^M
> Info: Sending request to instance number 1, port 2001
> Info: Trying to contact MPMall:1 on (2001)
> Info: attempting to connect to localhost on port 2001
> Info: MPMall:1 on (2001) connected [pooled: No]
> Info: Request GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html HTTP/1.1^M
>  sent, awaiting response
> Debug: ac_readConfiguration(): skipped reading config
> Info: New response: HTTP/1.1 302 Apple WebObjects
> 
> A failed request logged:
> 
> Info: received ->200 Apple
> Debug: <WebObjects Apache Module> new translate: 
> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html
> Info: <WebObjects Apache Module> new request: 
> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html
> Debug: App Name: 
> MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html
>  (6)
> Info: V4 URL: 
> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html
> Info: tr_selectInstance(): scheduler failed to select instance.
> Error: Request handling error: The requested application was not found on 
> this server.
> 
> Given these intermittent failures again, I thought I would start a couple of 
> new instances, but they failed to start, with no log output. All I had done 
> was to restart apache after configuring logging to debug.
> 
> Things are going a bit pear shaped.
> 
> John
> 
> On 27 Feb 2014, at 10:34, John Pollard <j...@pollardweb.com> wrote:
> 
> For the record, including:
> -_DeploymentDebugging true
> when running wotaskd didn't shed any light on why the apps don't start (no 
> log files)
> A change of SiteConfig.xml to an older auto-saved one didn't help, so that 
> seems fine
> Using ps to look for apps that wotaskd tries to start gives nothing, so they 
> don't seem to be launching at all, or too briefly to spot with no log output
> I can run the apps manually as the appserver user.
> Is there now way for wotaskd to pass on the stdout/stderrror from the apps it 
> tries to launch?
> 
> On 27 Feb 2014, at 09:38, John Pollard <j...@pollardweb.com> wrote:
> 
> Hi Chuck,
> 
> On 26 Feb 2014, at 22:26, Chuck Hill <ch...@global-village.net> wrote:
> 
> Hi John,
> 
> On 2/25/2014, 10:59 PM, "John Pollard" wrote:
> 
> Thanks for the feedback, really useful. To try to stop the intermittent 
> errors, I decided to restart wotaskd in this way:
> 
> sudo /etc/init.d/webobjects restart &
> 
> I think you can just kill it and let it get restarted automatically.  I 
> usually do this on OS X, but I think that init.d will do the same-ish thing.
> 
> I know this works on my Mac dev box, but I just tried it on a test deployment 
> server and it didn't restart after I killed it.
>  /etc/init.d/webobjects uses
> ps aux | awk '/WOPort 1085/ && !/awk/ {print $2}'
> to find the process and kills it
> 
> 
> 
> 
> Somehow this created a cycle of reported app deaths and restarts in 
> JavaMonitior which I only discovered in the early hours, though I don't see a 
> long trail of app log files, suggesting the restarts might not have been 
> happening in reality, or logs couldn't be written.
> 
> At first I thought more than one wotaskd was starting, but that can’t be as 
> there would be a port conflict.  Usually killing wotaskd and letting it 
> restart is clean and does not result in issues like this.
> 
> 
> 
> wotaskd and JavaMonitor run as appserver and I couldn't see a permissions 
> problem either in SiteConfig.xml or for the app log files
> 
> When a reboot resulted in the same problem (now really panic), I launched a 
> replacement live app server from a recent baseline and that was ok
> 
> That seems a little odd.  What had changed between the two?
> 
> No change, it was a recent baseline. The only points were that:
> - I had been seeing the intermittent: The requested application was not found 
> on this server.
> - I had tried to restart with sudo /etc/init.d/webobjects restart &
> (which I have subsequently shown seems to behave ok on another replicated 
> server)
> 
> 
> 
> One thought, can this ever be a problem?.... a WO server needs to start say 
> 20 apps which is a big load on the system, if too big a load, could wotaskd 
> think the apps failed to start and get into a cycle of restarting. I don't 
> think this is it, but a thought.
> 
> Yes, that can happen.  To mitigate this, set Time Allowed For Startup and 
> check Phased Startup in the Application Settings section of the Application 
> Config page.
> 
> Thanks, I do have Phased Startup and I have now doubled the startup times 
> from 30 to 60, though I don't think this is it. The key thing seems to be 
> that the apps weren't even getting to the point of producing any log output.
> 
> You are so helpful Chuck, thank you.
> John
> 
> 
> 
> Chuck
> 
> 
> 
> On 26 Feb 2014, at 00:07, Chuck Hill <ch...@global-village.net> wrote:
> 
> I usually set the apps to restart once a week.  I never restart wotaskd.
> 
> Chuck
> 
> On 2/25/2014, 7:09 AM, "John Pollard" wrote:
> 
> Do WO deployments / wotaskd need any occasional restarts scheduled? Our live 
> server uptime is currently 118 days.
> Might a wotaskd restart help with the intermittent "The requested application 
> was not found on this server."?
> 
> On 25 Feb 2014, at 11:50, John Pollard <j...@pollardweb.com> wrote:
> 
> Over the past few weeks we have started to notice the occasional: "The 
> requested application was not found on this server."
> 
> From experimenting, am I right in thinking this is from the apache WebObjects 
> module, because I am able to trigger this by stopping wotaskd?
> 
> Why might the WebObjects apache module not be able to find wotaskd, very 
> occasionally, if this is what is happening?
> 
> The wotaskd we are running is a Wonder version from March 2013.
> 
> The deployment is on Amazon Linux. No shortage of ram on the box and running 
> a fairly light load e.g. 20,000 transactions a day across 6 app instances, 
> averaging about 0.05 seconds per request with many seconds idle between 
> request.
> 
> The WO apache module used is: /usr/lib64/httpd/modules/mod_WebObjects.so
> -rwxr-xr-x 1 root root  395668 Apr 28  2011 mod_WebObjects.so
> 
> Apache version:
> Server version: Apache/2.2.25 (Unix)
> Server built:   Jul 12 2013 01:00:05
> 
> Any pointers as to how to track down the intermittent error would be a boost!
> 
> Thanks
> John
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com
> 
> This email sent to j...@pollardweb.com
> 
> 
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net
> 
> This email sent to ch...@global-village.net
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com
> 
> This email sent to jpoll...@inrax.com
> 
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com
> 
> This email sent to jpoll...@inrax.com
> 
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com
> 
> This email sent to j...@pollardweb.com
> 
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com
> 
> This email sent to j...@pollardweb.com
> 
> 
>  _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/probert%40macti.ca
> 
> This email sent to prob...@macti.ca
> 
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com
> 
> This email sent to jpoll...@inrax.com

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list      (Webobjects-deploy@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Reply via email to