I found the web server log was only written half of the day -- the half when crashes were not occurring. It seems the problem was my own CustomLog directives for apache on OS X Server. Though I followed the instructions from Apple to a "T", I think log lines went to /dev/null half the day.
Logging has been fixed now and upon another crash I'll check for a taf or a code block that might be causing the trouble. When I find out anything I'll post it to the list. Thanks for all your help, - Jeff (BTW, if you're running apache on OS X Server and are using your own custom log format, make sure you're getting 24 hours of log info in your access_log. Feel free to email me about this off-list.) >Unfortunately, these things don't show up on the mac log. > >Robert. > >On Wednesday, January 29, 2003, at 04:16 AM, Web Dude wrote: > >>Not sure about your log files, but in Windows if you pay attention to the threads >size on the tafs that are running, you may find the problem. I have found corrupt >tafs this way. Threads in the hundreds of thousands for an execution will indicate a >loop or a file in a corrupt state. Also look for any KILLS as in: >> >>KILLED E:blahblah/whatever/searchcustomers.taf >> >> >>>Jeff, >>> >>>You might get a clue on the last TAF served by corresponding the web server log >file with the Witango log file. That will always keep logging, even when the daemon >dies. If you can look at a few instances you might build a case for common culprits >to find specific files that could be causing the crash, as Robert mentioned. >>> >>>Jason >>> >>>On Tuesday, January 28, 2003, at 10:24 PM, Robert Garcia wrote: >>> >>>>Before you troubleshoot further, you have to determine if you have a specific >method/taf that is causing the problem. Unfortunately the OS X logging doesn't work >as well as on the windows version. >>>> >>>>Robert >>>>On Tuesday, January 28, 2003, at 06:17 PM, Jeffrey Bohmer wrote: >>>> >>>>> >>>>>The load is real traffic. The witangoevents.log does not specify a taf, >>>>>rather it says things like: >>>>> >>>>>[20794] 2003-01-28 10:23:02 RUNTIME INFO Started accepting user >>>>>requests >>>>>[20794] 2003-01-28 10:50:08 RUNTIME FATAL Caught fatal signal >>>>>SIGSEGV; thread id = 3801008 >>>>>[21067] 2003-01-28 10:51:00 START INFO Witango daemon started >>>>> >>>>> >>>>>The Witango.log does not contain anything about the crash. It also never >>>>>has a CRLF on the last line written, so the startup message does not appear >>>>>on it's own line. >>>>> >>>>>The server actually crashes and the witangod process dies. A watcher >>>>>script is called by cron every minute to check on the server and >>>>>restart it >>>>>if not running. >>>>> >>>>>I've set THREADPOOLSIZE=10 and LOGGINGLEVEL=1. When I get a chance to >>>>>restart the server, I'll watch it and see what happens. >>>>> >>>>>BTW, thanks for all your help!, >>>>>- Jeff >>>>> >>>>> >>>>>At 05:36 PM 1/28/03 -0800, you wrote: >>>>>>What are using to simulate the load? Are you hitting the same taf >>>>>>over >>>>>>and over? Are you monitoring real traffic? If you are monitoring >>>>>>realtraffic, does the witangoevents.log tell you which taf the app >>>>>>server crashed on? When it crashes, is it actually crashing, >>>>>>producing >>>>>>a crash log, or just hanging indefinitely? >>>>>> >>>>>>Robert. >>>>>> >>>>>>On Tuesday, January 28, 2003, at 12:57 PM, Jeffrey Bohmer wrote: >>>>>> >>>>>>> >>>>>>>I've been monitoring the system with top -u for a while. >>>>>>>witangod's >>>>>>>threads are always 3 above THREADPOOLSIZE. CPU usage for witangod is >>>>>>>usually 90 - 110% when the site is getting hits. When no hits are >>>>>>>coming in, it drops out of the list. (I assume CPU usage can go >>>>>>>above >>>>>>>100% because this is an SMP machine.) >>>>>>> >>>>>>>I will try out OpenLink's iODBC manager, if necessary. But I'd like >>>>>>>to try what I can with my current setup first. >>>>>>> >>>>>>>- Jeff >>>>>>> >>>>>>>>The threadpoolsize acts different than you think it would. I have >>>>>>>>done alot of testing, and the more you add, the slower, and more >>>>>>>>buggy the server gets. It is worse on the mac than windows, but >>>>>>>>occurs on both. Try setting the threadpoolsize to 10. Also, before >>>>>>>>you test, open the terminal, and enter: >>>>>>>> >>>>>>>>top -u 5 >>>>>>>> >>>>>>>>Then leave this running while under load. Monitor the CPU usage of >>>>>>>>the witango process, and also check the number of threads. It >>>>>>>>should >>>>>>>>be 2 or 3 above the threadpool size. Let me know what you see. >>>>>>>> >>>>>>>>Robert. >>>>>>>> >>>>>>>>On Tuesday, January 28, 2003, at 11:58 AM, Jeffrey Bohmer wrote: >>>>>>>> >>>>>>>>> >>>>>>>>>For a few minutes before the crashes, the app server gets about >>>>>>>>>35-45 requests per minute. Other hardware/config info: >>>>>>>>> >>>>>>>>> Dual 1.25GHz G4 >>>>>>>>> 1.75 GB memory >>>>>>>>> PostgreSQL has it's own drives for data & logs. >>>>>>>>> >>>>>>>>> THREADPOOLSIZE=25 >>>>>>>>> LOGGINGLEVEL=3 >>>>>>>>> >>>>>>>>> All caches/buffers are quite large and no swapping occurs. >>>>>>>>> >>>>>>>>> >>>>>>>>>Is THREADPOOLSIZE too low for this traffic? >>>>>>>>> >>>>>>>>>Also, I have logging turned up to see if a particular piece of >>>>>>>>>code >>>>>>>>>is being executed before the crash (or some kind of pattern). >>>>>>>>>But >>>>>>>>>if the last part of the log is lost on a crash, then I won't get >>>>>>>>>that info and could turn logging back down to 1. >>>>>>>>> >>>>>>>>>- Jeff >>>>>>>>> >>>>>>>>>>How heavy a load? I have tested it under heavy loads, and It did >>>>>>>>>>well, where it used to crash. There is still a point where it >>>>>>>>>>crashes when absolutely hammered, but it seems to be beyond the >>>>>>>>>>point of reasonable expectation. If the cache is off, it will >>>>>>>>>>crash under a lighter load. Looking at your setup, I would try >>>>>>>>>>the iODBC from openlink. You can get it from my sight if you wist >>>>>>>>>>at http://www.theradmac.com/ >>>>>>>>>> >>>>>>>>>>Robert Garcia >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>On Tuesday, January 28, 2003, at 11:23 AM, Jeffrey Bohmer wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>TRUE for both. (The CACHESIZE is about 20MB.) >>>>>>>>>>> >>>>>>>>>>>- Jeff >>>>>>>>>>> >>>>>>>>>>>>What are the: >>>>>>>>>>>> >>>>>>>>>>>>CACHE= >>>>>>>>>>>>CACHEINCLUDEFILES= >>>>>>>>>>>> >>>>>>>>>>>>variables set to in the witango.ini file? >>>>>>>>>>>> >>>>>>>>>>>>Robert Garcia >>>>>>>>>>>> >>>>>>>>>>>>On Tuesday, January 28, 2003, at 08:49 AM, Jeffrey Bohmer >>>>>>>>>>>>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>The latest production app server on OS X crashes every so >>>>>>>>>>>>>often, >>>>>>>>>>>>>when traffic is fairly high. The app server's FATAL signal is >>>>>>>>>>>>>either a bus error or segmentation violation, usually the >>>>>>>>>>>>>latter. >>>>>>>>>>>>>If anyone has any ideas as to why this might happen, or >>>>>>>>>>>>>things >>>>>>>>>>>>>to try to prevent it, I would be happy to receive your >>>>>>>>>>>>>insight. >>>>>>>>>>>>> >>>>>>>>>>>>>I wonder if some code is causing the crash. But I believe the >>>>>>>>>>>>>way the Witango.log is written to disk makes this difficult >>>>>>>>>>>>to >>>>>>>>>>>>>determine. It seems the app server delays writes to >>>>>>>>>>>>>Witango.log. >>>>>>>>>>>>>I bet the last bit of log info is lost when a crash occurs. >>>>>>>>>>>>>Therefore, I don't know what was happening just before a >>>>>>>>>>>>>crash. >>>>>>>>>>>>>Am I correct in my assumptions here? >>>>>>>>>>>>> >>>>>>>>>>>>>FYI ... in October, I added a bug report about this at >>>>>>>>>>>>>developer.witango.com. The beta server had this problem more >>>>>>>>>>>>>often >>>>>>>>>>>>>that the production server, but it's still a problem. Bug >>>>>>>>>>>>>Track >>>>>>>>>>>>>shows an empty status for this report (and another bug >>>>>>>>>>>>>report I >>>>>>>>>>>>>added). >>>>>>>>>>>>> >>>>>>>>>>>>>Witango App Server 5.0.1.054 >>>>>>>>>>>>>PostgreSQL 7.3.1 >>>>>>>>>>>>>OS X Server 10.2.3 >>>>>>>>>>>>>stock iODBC 2.1.6 >>>>>>>>>>>>>stock apache 1.3.27 >>>>>>>>>>>>> >>>>>>>>>>>>>Thank you, >>>>>>>>>>>>>- Jeff >>>>>>>>>>>>>-- Jeff Bohmer >>>>>>>>>>>>>VisionLink, Inc. >>>>>>>>>>>>>_________________________________ >>>>>>>>>>>>>303.402.0170 >>>>>>>>>>>>>www.visionlink.org >>>>>>>>>>>>>_________________________________ >>>>>>>>>>>>>People. Tools. Change. Community. >>>>>>>>>>>>>____________________________________________________________ __ ___ >>>>>>>>>>>>>__ __ __ _ >>>>>>>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>>>>> with unsubscribe witango-talk in the message body >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>-- >>>>>>>>>>>> >>>>>>>>>>>>Robert Garcia >>>>>>>>>>>>BigHead Technology >>>>>>>>>>>>2781 N Carlmont Pl >>>>>>>>>>>>Simi Valley, CA 93065 >>>>>>>>>>>>Phone 805.522.8577 >>>>>>>>>>>>http://www.bighead.net/ >>>>>>>>>>>>http://www.theradmac.com/ >>>>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>>>> >>>>>>>>>>>>_____________________________________________________________ __ ___ >>>>>>>>>>>>__ __ __ >>>>>>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>>>> with unsubscribe witango-talk in the message body >>>>>>>>>>> >>>>>>>>>>>-- >>>>>>>>>>> >>>>>>>>>>>Jeff Bohmer >>>>>>>>>>>VisionLink, Inc. >>>>>>>>>>>_________________________________ >>>>>>>>>>>303.402.0170 >>>>>>>>>>>www.visionlink.org >>>>>>>>>>>_________________________________ >>>>>>>>>>>People. Tools. Change. Community. >>>>>>>>>>>______________________________________________________________ __ ___ >>>>>>>>>>>__ __ _ >>>>>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>>> with unsubscribe witango-talk in the message body >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>-- >>>>>>>>>> >>>>>>>>>>Robert Garcia >>>>>>>>>>BigHead Technology >>>>>>>>>>2781 N Carlmont Pl >>>>>>>>>>Simi Valley, CA 93065 >>>>>>>>>>Phone 805.522.8577 >>>>>>>>>>http://www.bighead.net/ >>>>>>>>>>http://www.theradmac.com/ >>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>> >>>>>>>>>>_______________________________________________________________ __ ___ >>>>>>>>>>__ __ >>>>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>>>[EMAIL PROTECTED] >>>>>>>>>> with unsubscribe witango-talk in the message body >>>>>>>>> >>>>>>>>>-- >>>>>>>>> >>>>>>>>>Jeff Bohmer >>>>>>>>>VisionLink, Inc. >>>>>>>>>_________________________________ >>>>>>>>>303.402.0170 >>>>>>>>>www.visionlink.org >>>>>>>>>_________________________________ >>>>>>>>>People. Tools. Change. Community. >>>>>>>>>________________________________________________________________ __ ___ >>>>>>>>>__ _ >>>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>>[EMAIL PROTECTED] >>>>>>>>> with unsubscribe witango-talk in the message body >>>>>>>>> >>>>>>>> >>>>>>>>-- >>>>>>>> >>>>>>>>Robert Garcia >>>>>>>>BigHead Technology >>>>>>>>2781 N Carlmont Pl >>>>>>>>Simi Valley, CA 93065 >>>>>>>>Phone 805.522.8577 >>>>>>>>http://www.bighead.net/ >>>>>>>>http://www.theradmac.com/ >>>>>>>>[EMAIL PROTECTED] >>>>>>>> >>>>>>>>_________________________________________________________________ __ ___ >>>>>>>>__ >>>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>>[EMAIL PROTECTED] >>>>>>>> with unsubscribe witango-talk in the message body >>>>>>> >>>>>>>-- >>>>>>> >>>>>>>Jeff Bohmer >>>>>>>VisionLink, Inc. >>>>>>>_________________________________ >>>>>>>303.402.0170 >>>>>>>www.visionlink.org >>>>>>>_________________________________ >>>>>>>People. Tools. Change. Community. >>>>>>>__________________________________________________________________ __ ___ >>>>>>>_ >>>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>>[EMAIL PROTECTED] >>>>>>> with unsubscribe witango-talk in the message body >>>>>>> >>>>>> >>>>>>-- >>>>>> >>>>>>Robert Garcia >>>>>>BigHead Technology >>>>>>2781 N Carlmont Pl >>>>>>Simi Valley, CA 93065 >>>>>>Phone 805.522.8577 >>>>>>http://www.bighead.net/ >>>>>>http://www.theradmac.com/ >>>>>>[EMAIL PROTECTED] >>>>>> >>>>>>___________________________________________________________________ __ ___ >>>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>>[EMAIL PROTECTED] >>>>>> with unsubscribe witango-talk in the message body >>>>>> >>>>> >>>>>-------------------------- >>>>> bohmer >>>>> Mobility Productions >>>>> http://mobility303.com >>>>> phone: 303.516.9544 >>>>>-------------------------- >>>>>____________________________________________________________________ __ __ >>>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>>[EMAIL PROTECTED] >>>>> with unsubscribe witango-talk in the message body >>>>> >>>> >>>>-- >>>> >>>>Robert Garcia >>>>BigHead Technology >>>>2781 N Carlmont Pl >>>>Simi Valley, CA 93065 >>>>Phone 805.522.8577 >>>>http://www.bighead.net/ >>>>http://www.theradmac.com/ >>>>[EMAIL PROTECTED] >>>> >>>>_____________________________________________________________________ __ _ >>>>TO UNSUBSCRIBE: send a plain text/US ASCII email to >>>>[EMAIL PROTECTED] >>>> with unsubscribe witango-talk in the message body >>>> >>>-- >>>____________________________________________________________________ >>> >>>Jason Pamental, President >>> [EMAIL PROTECTED] >>> >>>Bathysphere Digital Media Services, Inc. >>>http://bathyspheredms.com >>>____________________________________________________________________ >>> >>>Tel: 401.490.6830 Fax: 401.490.6831 >>>________________________________________ >>> >>> >>>A North American Distributor for Witango (http://www.witango.us) >>> >>>Rapid Web Application Development - XML Execution Engine >>> >>> >>>______________________________________________________________________ __ >>>TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] >>> with unsubscribe witango-talk in the message body >> >> >>-- >>_______________________________________________________________________ _ >>TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] >> with unsubscribe witango-talk in the message body >> > >-- > >Robert Garcia >BigHead Technology >2781 N Carlmont Pl >Simi Valley, CA 93065 >Phone 805.522.8577 >http://www.bighead.net/ >http://www.theradmac.com/ >[EMAIL PROTECTED] > >________________________________________________________________________ >TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] > with unsubscribe witango-talk in the message body -- Jeff Bohmer VisionLink, Inc. _________________________________ 303.402.0170 www.visionlink.org _________________________________ People. Tools. Change. Community. ________________________________________________________________________ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body
