John/Chuck, I have given http://www.inrax.com/cgi-bin/WebObjects/WOAdaptorInfo public access; is there anything on there that stands out?
Also, the debugging John mentioned: WebObjectsLog /Library/WebObjects/Logs/WebObjects.log Debug I already have that and it is what I have been cutting and pasting, but isn't that the WO apache module rather than wotaskd? If I stop wotaskd completely, then I can guarantee the "The requested application was not found on this server." error, so my assumption is that for whatever reason the WO apache module (WO adapter?) can't talk to wotaskd, at least not 100% reliably. Or have I got it wrong? What I can't get is logging of the decisions on which instance to pick next in routing by wotaskd; if I could get that I would then see if the request is getting to wotaskd from the WO adaptor. Many thanks again. John On 27 Feb 2014, at 18:13, John Huss <johnth...@gmail.com> wrote: > You probably DO want the adaptor logging on per > /System/Library/WebObjects/Adaptors/Apache2.2/apache.conf > > # Note: To enable logging, touch '/tmp/logWebObjects' as the administrator > user (usually root). > # > # The following line is the default: > WebObjectsLog /Library/WebObjects/Logs/WebObjects.log Debug > > > Also, you can inspect some of the adaptor's state by going to: > > http://yourhost/cgi-bin/WebObjects/WOAdaptorInfo?yourPasswordInTheApacheConfFileMentionedAbove > > In particular look at the "dead timeout" on that page. > > > > On Thu, Feb 27, 2014 at 11:52 AM, John Pollard <j...@pollardweb.com> wrote: > Back to my original problem, these errors are happening about one every 100 > requests: > > Info: <WebObjects Apache Module> new request: > /cgi-bin/WebObjects/MPMall.woa/wa/1/63/2392-Gilets-SALE.html > Debug: App Name: MPMall.woa/wa/1/63/2392-Gilets-SALE.html (6) > Info: V4 URL: /cgi-bin/WebObjects/MPMall.woa/wa/1/63/2392-Gilets-SALE.html > Info: tr_selectInstance(): scheduler failed to select instance. > Error: Request handling error: The requested application was not found on > this server. > > I can't see logging from wotaskd in terms of making its choices about which > instance is next, even for the working requests. I tried passing > -_DeploymentDebugging true, when running wotaskd but the only logging I see > is Lifebeats coming in. Am I misunderstanding what wotaskd does? > > The request never makes it to the application. Would there be much point in > be rebuilding the latest Wonder wotaskd? The one I have running is about a > year old. Or could the WO apache module be at fault, my mod_WebObjects.so > file is dated 38 April 2011, apache v. 2.2.25 > > Many thanks again, > John > > On 27 Feb 2014, at 16:42, John Pollard <j...@pollardweb.com> wrote: > >> John, a big thank you as this seems to have answered why I couldn't start >> apps at all. >> I don't have the LOG path shown, so changed it to one I use for logging and >> now can fire up apps again. >> Or removing /tmp/logWebObjects also fixes it. >> Perhaps this script should test for access to the LOG file and if no access, >> not launch the apps that way. >> Many thanks; I don't think this relates to my underlying intermittent "The >> requested application was not found on this server." but it brings a bit of >> sanity back! >> John >> >> On 27 Feb 2014, at 15:20, John Huss <johnth...@gmail.com> wrote: >> >>> If you're using Wonder's wotaskd (or care to replace the following script) >>> you can turn on logging for instance startup in: >>> >>> /Library/WebObjects/JavaApplications/wotaskd.woa/Contents/Resources/SpawnOfWotaskd.sh >>> >>> #!/bin/sh >>> >>> # To enable logging of instance startup run the command 'touch >>> /tmp/logWebObjects' >>> >>> # Log messages will be written to: >>> LOG=/Library/WebObjects/Logs/SpawnOfWotaskd.log >>> >>> if [ -f /tmp/logWebObjects ]; then >>> >>> echo "************" >>${LOG} >>> echo "date: `date`" >>${LOG} >>> echo "args: $@" >>${LOG} >>> $@ 1>>${LOG} 2>&1 & >>> >>> else >>> >>> $@ 1>/dev/null 2>&1 & >>> >>> fi >>> >>> >>> On Thu, Feb 27, 2014 at 8:34 AM, John Pollard <j...@pollardweb.com> wrote: >>> I am getting more intermittent app not found on server. >>> Looking at the WO apache module log: >>> >>> A good request sequence logged looks like this: >>> Info: received ->200 Apple >>> Debug: <WebObjects Apache Module> new translate: >>> /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: <WebObjects Apache Module> translate - DECLINED: >>> /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: WebObjects_handler declined! /ires/8206/GOHWeb/miscRes/gohStyle11.css >>> Debug: <WebObjects Apache Module> new translate: /favicon.ico >>> Debug: <WebObjects Apache Module> translate - DECLINED: /favicon.ico >>> Debug: WebObjects_handler declined! /favicon.ico >>> Info: New request is GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html >>> HTTP/1.1^M >>> Info: Sending request to instance number 1, port 2001 >>> Info: Trying to contact MPMall:1 on (2001) >>> Info: attempting to connect to localhost on port 2001 >>> Info: MPMall:1 on (2001) connected [pooled: No] >>> Info: Request GET /cgi-bin/WebObjects/MPMall.woa/1/wa/8/43.html HTTP/1.1^M >>> sent, awaiting response >>> Debug: ac_readConfiguration(): skipped reading config >>> Info: New response: HTTP/1.1 302 Apple WebObjects >>> >>> A failed request logged: >>> >>> Info: received ->200 Apple >>> Debug: <WebObjects Apache Module> new translate: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Info: <WebObjects Apache Module> new request: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Debug: App Name: >>> MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> (6) >>> Info: V4 URL: >>> /cgi-bin/WebObjects/MPMall.woa/wa/2/52/3689-Garden-Sprinklers/106568-DOG-SPRINKLER-BRONZE-FINISH.html >>> Info: tr_selectInstance(): scheduler failed to select instance. >>> Error: Request handling error: The requested application was not found on >>> this server. >>> >>> Given these intermittent failures again, I thought I would start a couple >>> of new instances, but they failed to start, with no log output. All I had >>> done was to restart apache after configuring logging to debug. >>> >>> Things are going a bit pear shaped. >>> >>> John >>> >>> On 27 Feb 2014, at 10:34, John Pollard <j...@pollardweb.com> wrote: >>> >>>> For the record, including: >>>> -_DeploymentDebugging true >>>> when running wotaskd didn't shed any light on why the apps don't start (no >>>> log files) >>>> A change of SiteConfig.xml to an older auto-saved one didn't help, so that >>>> seems fine >>>> Using ps to look for apps that wotaskd tries to start gives nothing, so >>>> they don't seem to be launching at all, or too briefly to spot with no log >>>> output >>>> I can run the apps manually as the appserver user. >>>> Is there now way for wotaskd to pass on the stdout/stderrror from the apps >>>> it tries to launch? >>>> >>>> On 27 Feb 2014, at 09:38, John Pollard <j...@pollardweb.com> wrote: >>>> >>>>> Hi Chuck, >>>>> >>>>> On 26 Feb 2014, at 22:26, Chuck Hill <ch...@global-village.net> wrote: >>>>> >>>>>> Hi John, >>>>>> >>>>>> On 2/25/2014, 10:59 PM, "John Pollard" wrote: >>>>>> >>>>>> Thanks for the feedback, really useful. To try to stop the intermittent >>>>>> errors, I decided to restart wotaskd in this way: >>>>>> >>>>>> sudo /etc/init.d/webobjects restart & >>>>>> >>>>>> I think you can just kill it and let it get restarted automatically. I >>>>>> usually do this on OS X, but I think that init.d will do the same-ish >>>>>> thing. >>>>> >>>>> I know this works on my Mac dev box, but I just tried it on a test >>>>> deployment server and it didn't restart after I killed it. >>>>> /etc/init.d/webobjects uses >>>>> ps aux | awk '/WOPort 1085/ && !/awk/ {print $2}' >>>>> to find the process and kills it >>>>> >>>>>> >>>>>> >>>>>> Somehow this created a cycle of reported app deaths and restarts in >>>>>> JavaMonitior which I only discovered in the early hours, though I don't >>>>>> see a long trail of app log files, suggesting the restarts might not >>>>>> have been happening in reality, or logs couldn't be written. >>>>>> >>>>>> At first I thought more than one wotaskd was starting, but that can’t be >>>>>> as there would be a port conflict. Usually killing wotaskd and letting >>>>>> it restart is clean and does not result in issues like this. >>>>>> >>>>>> >>>>>> >>>>>> wotaskd and JavaMonitor run as appserver and I couldn't see a >>>>>> permissions problem either in SiteConfig.xml or for the app log files >>>>>> >>>>>> When a reboot resulted in the same problem (now really panic), I >>>>>> launched a replacement live app server from a recent baseline and that >>>>>> was ok >>>>>> >>>>>> That seems a little odd. What had changed between the two? >>>>> >>>>> No change, it was a recent baseline. The only points were that: >>>>> - I had been seeing the intermittent: The requested application was not >>>>> found on this server. >>>>> - I had tried to restart with sudo /etc/init.d/webobjects restart & >>>>> (which I have subsequently shown seems to behave ok on another replicated >>>>> server) >>>>> >>>>>> >>>>>> >>>>>> One thought, can this ever be a problem?.... a WO server needs to start >>>>>> say 20 apps which is a big load on the system, if too big a load, could >>>>>> wotaskd think the apps failed to start and get into a cycle of >>>>>> restarting. I don't think this is it, but a thought. >>>>>> >>>>>> Yes, that can happen. To mitigate this, set Time Allowed For Startup >>>>>> and check Phased Startup in the Application Settings section of the >>>>>> Application Config page. >>>>> >>>>> Thanks, I do have Phased Startup and I have now doubled the startup times >>>>> from 30 to 60, though I don't think this is it. The key thing seems to be >>>>> that the apps weren't even getting to the point of producing any log >>>>> output. >>>>> >>>>> You are so helpful Chuck, thank you. >>>>> John >>>>> >>>>>> >>>>>> >>>>>> Chuck >>>>>> >>>>>> >>>>>> >>>>>> On 26 Feb 2014, at 00:07, Chuck Hill <ch...@global-village.net> wrote: >>>>>> >>>>>>> I usually set the apps to restart once a week. I never restart wotaskd. >>>>>>> >>>>>>> Chuck >>>>>>> >>>>>>> On 2/25/2014, 7:09 AM, "John Pollard" wrote: >>>>>>> >>>>>>> Do WO deployments / wotaskd need any occasional restarts scheduled? Our >>>>>>> live server uptime is currently 118 days. >>>>>>> Might a wotaskd restart help with the intermittent "The requested >>>>>>> application was not found on this server."? >>>>>>> >>>>>>> On 25 Feb 2014, at 11:50, John Pollard <j...@pollardweb.com> wrote: >>>>>>> >>>>>>> Over the past few weeks we have started to notice the occasional: "The >>>>>>> requested application was not found on this server." >>>>>>> From experimenting, am I right in thinking this is from the apache >>>>>>> WebObjects module, because I am able to trigger this by stopping >>>>>>> wotaskd? >>>>>>> Why might the WebObjects apache module not be able to find wotaskd, >>>>>>> very occasionally, if this is what is happening? >>>>>>> The wotaskd we are running is a Wonder version from March 2013. >>>>>>> The deployment is on Amazon Linux. No shortage of ram on the box and >>>>>>> running a fairly light load e.g. 20,000 transactions a day across 6 app >>>>>>> instances, averaging about 0.05 seconds per request with many seconds >>>>>>> idle between request. >>>>>>> The WO apache module used is: /usr/lib64/httpd/modules/mod_WebObjects.so >>>>>>> -rwxr-xr-x 1 root root 395668 Apr 28 2011 mod_WebObjects.so >>>>>>> Apache version: >>>>>>> Server version: Apache/2.2.25 (Unix) >>>>>>> Server built: Jul 12 2013 01:00:05 >>>>>>> Any pointers as to how to track down the intermittent error would be a >>>>>>> boost! >>>>>>> Thanks >>>>>>> John >>>>>>> _______________________________________________ >>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>>>>>> This email sent to j...@pollardweb.com >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>> https://lists.apple.com/mailman/options/webobjects-deploy/chill%40global-village.net >>>>>>> >>>>>>> This email sent to ch...@global-village.net >>>>>>> _______________________________________________ >>>>>>> Do not post admin requests to the list. They will be ignored. >>>>>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>>>>> Help/Unsubscribe/Update your Subscription: >>>>>>> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >>>>>>> >>>>>>> This email sent to jpoll...@inrax.com >>>>>> >>>>>> _______________________________________________ >>>>>> Do not post admin requests to the list. They will be ignored. >>>>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>>>> Help/Unsubscribe/Update your Subscription: >>>>>> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >>>>>> >>>>>> This email sent to jpoll...@inrax.com >>>>> >>>>> _______________________________________________ >>>>> Do not post admin requests to the list. They will be ignored. >>>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>>> Help/Unsubscribe/Update your Subscription: >>>>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>>>> >>>>> This email sent to j...@pollardweb.com >>>> >>>> _______________________________________________ >>>> Do not post admin requests to the list. They will be ignored. >>>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>>> Help/Unsubscribe/Update your Subscription: >>>> https://lists.apple.com/mailman/options/webobjects-deploy/john%40pollardweb.com >>>> >>>> This email sent to j...@pollardweb.com >>> >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> https://lists.apple.com/mailman/options/webobjects-deploy/johnthuss%40gmail.com >>> >>> This email sent to johnth...@gmail.com >>> >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >>> Help/Unsubscribe/Update your Subscription: >>> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >>> >>> This email sent to jpoll...@inrax.com >> >> _______________________________________________ >> Do not post admin requests to the list. They will be ignored. >> Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) >> Help/Unsubscribe/Update your Subscription: >> https://lists.apple.com/mailman/options/webobjects-deploy/jpollard%40inrax.com >> >> This email sent to jpoll...@inrax.com > > > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/webobjects-deploy/johnthuss%40gmail.com > > This email sent to johnth...@gmail.com >
_______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-deploy mailing list (Webobjects-deploy@lists.apple.com) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com This email sent to arch...@mail-archive.com
