On Thu, Feb 27, 2014 at 1:59 PM, John Pollard <j...@pollardweb.com> wrote:
> John/Chuck, > > I have given http://www.inrax.com/cgi-bin/WebObjects/WOAdaptorInfo public > access; is there anything on there that stands out? > > Also, the debugging John mentioned: WebObjectsLog > /Library/WebObjects/Logs/WebObjects.log Debug > I already have that and it is what I have been cutting and pasting, but > isn't that the WO apache module rather than wotaskd? > If I stop wotaskd completely, then I can guarantee the "The requested > application was not found on this server." error, so my assumption is that > for whatever reason the WO apache module (WO adapter?) can't talk to > wotaskd, at least not 100% reliably. Or have I got it wrong? > Right, or the instances themselves can't talk to wotaskd reliably or they are deadlocked or overloaded somehow. > > What I can't get is logging of the decisions on which instance to pick > next in routing by wotaskd; if I could get that I would then see if the > request is getting to wotaskd from the WO adaptor. > The WO adaptor does the load balancing based on the settings in JavaMonitor, which by default is Round Robin I think (which is what you have). It doesn't matter, but a newer adaptor would also give you the option for Load Average which balances by looking at the active session count. > > Many thanks again. > > John > > On 27 Feb 2014, at 18:13, John Huss <johnth...@gmail.com> wrote: > > You probably DO want the adaptor logging on > per /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf > > # Note: To enable logging, touch '/tmp/logWebObjects' as the administrator > user (usually root). > # > # The following line is the default: > WebObjectsLog /Library/WebObjects/Logs/WebObjects.log Debug > > > Also, you can inspect some of the adaptor's state by going to: > > > http://yourhost/cgi-bin/WebObjects/WOAdaptorInfo?yourPasswordInTheApacheConfFileMentionedAbove > > In particular look at the "dead timeout" on that page. > > > > On Thu, Feb 27, 2014 at 11:52 AM, John Pollard <j...@pollardweb.com>wrote: > >> Back to my original problem, these errors are happening about one every >> 100 requests: >> >> Info: <WebObjects Apache Module> new request: >> /cgi-bin/WebObjects/MPMall.woa/wa/1/63/2392-Gilets-SALE.html >> Debug: App Name: MPMall.woa/wa/1/63/2392-Gilets-SALE.html (6) >> Info: V4 URL: /cgi-bin/WebObjects/MPMall.woa/wa/1/63/2392-Gilets-SALE.html >> Info: tr_selectInstance(): scheduler failed to select instance. >> Error: Request handling error: The requested application was not found >> on this server. >> >> I can't see logging from wotaskd in terms of making its choices about >> which instance is next, even for the working requests. I tried >> passing -_DeploymentDebugging true, when running wotaskd but the only >> logging I see is Lifebeats coming in. Am I misunderstanding what wotaskd >> does? >> >> The request never makes it to the application. Would there be much point >> in be rebuilding the latest Wonder wotaskd? The one I have running is about >> a year old. Or could the WO apache module be at fault, my mod_WebObjects.so >> file is dated 38 April 2011, apache v. 2.2.25 >> >> Many thanks again, >> John >> >> On 27 Feb 2014, at 16:42, John Pollard <j...@pollardweb.com> wrote: >> >> John, a big thank you as this seems to have answered why I couldn't start >> apps at all. >> I don't have the LOG path shown, so changed it to one I use for logging >> and now can fire up apps again. >> Or removing /tmp/logWebObjects also fixes it. >> Perhaps this script should test for access to the LOG file and if no >> access, not launch the apps that way. >> Many thanks; I don't think this relates to my underlying intermittent >> "The requested application was not found on this server." but it brings a >> bit of sanity back! >> John >> >> On 27 Feb 2014, at 15:20, John Huss <johnth...@gmail.com> wrote: >> >> If you're using Wonder's wotaskd (or care to replace the following >> script) you can turn on logging for instance startup in: >> >> >> /Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/SpawnOfWotaskd.sh >> >> #!/bin/sh >> >> # To enable logging of instance startup run the command 'touch >> /tmp/logWebObjects' >> >> # Log messages will be written to: >> LOG=/Library/WebObjects/Logs/SpawnOfWotaskd.log >> >> if [ -f /tmp/logWebObjects ]; then >> >> echo "************" >>${LOG} >> echo "date: `date`" >>${LOG} >> echo "args: $@" >>${LOG} >> $@ 1>>${LOG} 2>&1 & >> >> else >> >> $@ 1>/dev/null 2>&1 & >> >> fi >> >> >> On Thu, Feb 27, 2014 at 8:34 AM, John Pollard <j...@pollardweb.com>wrote: >> >>> I am getting more intermittent app not found on server. >>> Looking at the WO apache module log: >>> >>> A good request sequence logged looks like this: >>> Info: received ->200 Apple >>> Debug: <WebObjects Apache Module> new translate: >>> /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: <WebObjects Apache Module> translate - DECLINED: >>> /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: WebObjects_handler declined! >>> /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: <WebObjects Apache Module> new translate: /favicon.ico >>> Debug: <WebObjects Apache Module> translate - DECLINED: /favicon.ico >>> Debug: WebObjects_handler declined! /favicon.ico >>> Info: New request is GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html >>> HTTP/1.1^M >>> Info: Sending request to instance number 1, port 2001 >>> Info: Trying to contact MPMall:1 on (2001) >>> Info: attempting to connect to localhost on port 2001 >>> Info: MPMall:1 on (2001) connected [pooled: No] >>> Info: Request GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html HTTP/1.1 >>> ^M >>> sent, awaiting response >>> Debug: ac_readConfiguration(): skipped reading config >>> Info: New response: HTTP/1.1 302 Apple WebObjects >>> >>> A failed request logged: >>> >>> Info: received ->200 Apple >>> Debug: <WebObjects Apache Module> new translate: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Info: <WebObjects Apache Module> new request: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Debug: App Name: >>> MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> (6) >>> Info: V4 URL: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Info: tr_selectInstance(): scheduler failed to select instance. >>> Error: Request handling error: The requested application was not found >>> on this server. >>> >>> Given these intermittent failures again, I thought I would start a >>> couple of new instances, but they failed to start, with no log output. All >>> I had done was to restart apache after configuring logging to debug. >>> >>> Things are going a bit pear shaped. >>> >>> John >>> >>> On 27 Feb 2014, at 10:34, John Pollard <j...@pollardweb.com> wrote: >>> >>> For the record, including: >>> -_DeploymentDebugging true >>> when running wotaskd didn't shed any light on why the apps don't start >>> (no log files) >>> A change of SiteConfig.xml to an older auto-saved one didn't help, so >>> that seems fine >>> Using ps to look for apps that wotaskd tries to start gives nothing, so >>> they don't seem to be launching at all, or too briefly to spot with no log >>> output >>> I can run the apps manually as the appserver user. >>> Is there now way for wotaskd to pass on the stdout/stderrror from the >>> apps it tries to launch? >>> >>> On 27 Feb 2014, at 09:38, John Pollard <j...@pollardweb.com> wrote: >>> >>> Hi Chuck, >>> >>> On 26 Feb 2014, at 22:26, Chuck Hill <ch...@global-village.net> wrote: >>> >>> Hi John, >>> >>> On 2/25/2014, 10:59 PM, "John Pollard" wrote: >>> >>> Thanks for the feedback, really useful. To try to stop the >>> intermittent errors, I decided to restart wotaskd in this way: >>> >>> sudo /etc/init.d/webobjects restart & >>> >>> >>> I think you can just kill it and let it get restarted automatically. >>> I usually do this on OS X, but I think that init.d will do the same-ish >>> thing. >>> >>> >>> I know this works on my Mac dev box, but I just tried it on a test >>> deployment server and it didn't restart after I killed it. >>> /etc/init.d/webobjects uses >>> ps aux | awk '/WOPort 1085/ && !/awk/ {print $2}' >>> to find the process and kills it >>> >>> >>> >>> Somehow this created a cycle of reported app deaths and restarts in >>> JavaMonitior which I only discovered in the early hours, though I don't see >>> a long trail of app log files, suggesting the restarts might not have been >>> happening in reality, or logs couldn't be written. >>> >>> >>> At first I thought more than one wotaskd was starting, but that can't >>> be as there would be a port conflict. Usually killing wotaskd and letting >>> it restart is clean and does not result in issues like this. >>> >>> >>> >>> wotaskd and JavaMonitor run as appserver and I couldn't see a >>> permissions problem either in SiteConfig.xml or for the app log files >>> >>> When a reboot resulted in the same problem (now really panic), I >>> launched a replacement live app server from a recent baseline and that was >>> ok >>> >>> >>> That seems a little odd. What had changed between the two? >>> >>> >>> No change, it was a recent baseline. The only points were that: >>> - I had been seeing the intermittent: The requested application was not >>> found on this server. >>> - I had tried to restart with sudo /etc/init.d/webobjects restart & >>> (which I have subsequently shown seems to behave ok on another >>> replicated server) >>> >>> >>> >>> One thought, can this ever be a problem?.... a WO server needs to >>> start say 20 apps which is a big load on the system, if too big a load, >>> could wotaskd think the apps failed to start and get into a cycle of >>> restarting. I don't think this is it, but a thought. >>> >>> >>> Yes, that can happen. To mitigate this, set Time Allowed For Startup >>> and check Phased Startup in the Application Settings section of >>> the Application Config page. >>> >>> >>> Thanks, I do have Phased Startup and I have now doubled the startup >>> times from 30 to 60, though I don't think this is it. The key thing seems >>> to be that the apps weren't even getting to the point of producing any log >>> output. >>> >>> You are so helpful Chuck, thank you. >>> John >>> >>> >>> >>> Chuck >>> >>> >>> >>> On 26 Feb 2014, at 00:07, Chuck Hill <ch...@global-village.net> wrote: >>> >>> I usually set the apps to restart once a week. I never restart >>> wotaskd. >>> >>> Chuck >>> >>> On 2/25/2014, 7:09 AM, "John Pollard" wrote: >>> >>> Do WO deployments / wotaskd need any occasional restarts scheduled? >>> Our live server uptime is currently 118 days. >>> Might a wotaskd restart help with the intermittent "The requested >>> application was not found on this server."? >>> >>> On 25 Feb 2014, at 11:50, John Pollard <j...@pollardweb.com> wrote: >>> >>> Over the past few weeks we have started to notice the occasional: "The >>> requested application was not found on this server." >>> From experimenting, am I right in thinking this is from the apache >>> WebObjects module, because I am able to trigger this by stopping wotaskd? >>> Why might the WebObjects apache module not be able to find wotaskd, >>> very occasionally, if this is what is happening? >>> The wotaskd we are running is a Wonder version from March 2013. >>> The deployment is on Amazon Linux. No shortage of ram on the box and >>> running a fairly light load e.g. 20,000 transactions a day across 6 app >>> instances, averaging about 0.05 seconds per request with many seconds idle >>> between request. >>> The WO apache module used is: >>> /usr/lib64/httpd/modules/mod_WebObjects.so >>> -rwxr-xr-x 1 root root 395668 Apr 28 2011 mod_WebObjects.so >>> Apache version: >>> Server version: Apache/2.2.25 (Unix) >>> Server built: Jul 12 2013 01:00:05 >>> Any pointers as to how to track down the intermittent error would be a >>> boost! >>> Thanks >>> John >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>> This email sent to j...@pollardweb.com >>> >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net >>> >>> This email sent to ch...@global-village.net >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >>> >>> This email sent to jpoll...@inrax.com >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >>> >>> This email sent to jpoll...@inrax.com >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>> >>> This email sent to j...@pollardweb.com >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>> >>> This email sent to j...@pollardweb.com >>> >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> >>> https://lists.apple.com/mailman/options/webobjects-deploy/johnthuss%40gmail.com >>> >>> This email sent to johnth...@gmail.com >>> >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> >> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >> >> This email sent to jpoll...@inrax.com >> >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> >> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >> >> This email sent to jpoll...@inrax.com >> >> >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> >> https://lists.apple.com/mailman/options/webobjects-deploy/johnthuss%40gmail.com >> >> This email sent to johnth...@gmail.com >> > > > > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > > https://lists.apple.com/mailman/options/webobjects-deploy/johnthuss%40gmail.com > > This email sent to johnth...@gmail.com >
_______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com This email sent to arch...@mail-archive.com
