KILLED E:blahblah/whatever/searchcustomers.taf
Jeff,
You might get a clue on the last TAF served by corresponding the web server log file with the Witango log file. That will always keep logging, even when the daemon dies. If you can look at a few instances you might build a case for common culprits to find specific files that could be causing the crash, as Robert mentioned.
Jason
On Tuesday, January 28, 2003, at 10:24 PM, Robert Garcia wrote:
Before you troubleshoot further, you have to determine if you have a specific method/taf that is causing the problem. Unfortunately the OS X logging doesn't work as well as on the windows version.--
Robert
On Tuesday, January 28, 2003, at 06:17 PM, Jeffrey Bohmer wrote:
The load is real traffic. The witangoevents.log does not specify a taf, rather it says things like:[20794] 2003-01-28 10:23:02 RUNTIME INFO Started accepting user requests [20794] 2003-01-28 10:50:08 RUNTIME FATAL Caught fatal signal SIGSEGV; thread id = 3801008 [21067] 2003-01-28 10:51:00 START INFO Witango daemon started The Witango.log does not contain anything about the crash. It also never has a CRLF on the last line written, so the startup message does not appear on it's own line. The server actually crashes and the witangod process dies. A watcher script is called by cron every minute to check on the server and restart it if not running. I've set THREADPOOLSIZE=10 and LOGGINGLEVEL=1. When I get a chance to restart the server, I'll watch it and see what happens. BTW, thanks for all your help!, - Jeff At 05:36 PM 1/28/03 -0800, you wrote:What are using to simulate the load? Are you hitting the same taf over and over? Are you monitoring real traffic? If you are monitoring realtraffic, does the witangoevents.log tell you which taf the app server crashed on? When it crashes, is it actually crashing, producing a crash log, or just hanging indefinitely? Robert. On Tuesday, January 28, 2003, at 12:57 PM, Jeffrey Bohmer wrote:I've been monitoring the system with top -u for a while. witangod's threads are always 3 above THREADPOOLSIZE. CPU usage for witangod is usually 90 - 110% when the site is getting hits. When no hits are coming in, it drops out of the list. (I assume CPU usage can go above 100% because this is an SMP machine.) I will try out OpenLink's iODBC manager, if necessary. But I'd like to try what I can with my current setup first. - JeffThe threadpoolsize acts different than you think it would. I have done alot of testing, and the more you add, the slower, and more buggy the server gets. It is worse on the mac than windows, but occurs on both. Try setting the threadpoolsize to 10. Also, before you test, open the terminal, and enter: top -u 5 Then leave this running while under load. Monitor the CPU usage of the witango process, and also check the number of threads. It should be 2 or 3 above the threadpool size. Let me know what you see. Robert. On Tuesday, January 28, 2003, at 11:58 AM, Jeffrey Bohmer wrote:
For a few minutes before the crashes, the app server gets about
35-45 requests per minute. Other hardware/config info:
Dual 1.25GHz G4
1.75 GB memory
PostgreSQL has it's own drives for data & logs.
THREADPOOLSIZE=25
LOGGINGLEVEL=3
All caches/buffers are quite large and no swapping occurs.
Is THREADPOOLSIZE too low for this traffic?
Also, I have logging turned up to see if a particular piece of code
is being executed before the crash (or some kind of pattern). But
if the last part of the log is lost on a crash, then I won't get
that info and could turn logging back down to 1.
- Jeff
How heavy a load? I have tested it under heavy loads, and It did well, where it used to crash. There is still a point where it crashes when absolutely hammered, but it seems to be beyond the point of reasonable expectation. If the cache is off, it will crash under a lighter load. Looking at your setup, I would try the iODBC from openlink. You can get it from my sight if you wist at http://www.theradmac.com/ Robert Garcia On Tuesday, January 28, 2003, at 11:23 AM, Jeffrey Bohmer wrote:TRUE for both. (The CACHESIZE is about 20MB.) - JeffWhat are the: CACHE= CACHEINCLUDEFILES= variables set to in the witango.ini file? Robert Garcia On Tuesday, January 28, 2003, at 08:49 AM, Jeffrey Bohmer wrote:The latest production app server on OS X crashes every so often, when traffic is fairly high. The app server's FATAL signal is either a bus error or segmentation violation, usually the latter. If anyone has any ideas as to why this might happen, or things to try to prevent it, I would be happy to receive your insight. I wonder if some code is causing the crash. But I believe the way the Witango.log is written to disk makes this difficulttodetermine. It seems the app server delays writes to Witango.log. I bet the last bit of log info is lost when a crash occurs. Therefore, I don't know what was happening just before a crash. Am I correct in my assumptions here? FYI ... in October, I added a bug report about this at developer.witango.com. The beta server had this problem more often that the production server, but it's still a problem. Bug Track shows an empty status for this report (and another bug report I added). Witango App Server 5.0.1.054 PostgreSQL 7.3.1 OS X Server 10.2.3 stock iODBC 2.1.6 stock apache 1.3.27 Thank you, - Jeff -- Jeff Bohmer VisionLink, Inc. _________________________________ 303.402.0170 www.visionlink.org _________________________________ People. Tools. Change. Community. ______________________________________________________________ ___ __ __ __ _ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Robert Garcia BigHead Technology 2781 N Carlmont Pl Simi Valley, CA 93065 Phone 805.522.8577 http://www.bighead.net/ http://www.theradmac.com/ [EMAIL PROTECTED] _______________________________________________________________ ___ __ __ __ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Jeff Bohmer VisionLink, Inc. _________________________________ 303.402.0170 www.visionlink.org _________________________________ People. Tools. Change. Community. ________________________________________________________________ ___ __ __ _ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Robert Garcia BigHead Technology 2781 N Carlmont Pl Simi Valley, CA 93065 Phone 805.522.8577 http://www.bighead.net/ http://www.theradmac.com/ [EMAIL PROTECTED] _________________________________________________________________ ___ __ __ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Jeff Bohmer VisionLink, Inc. _________________________________ 303.402.0170 www.visionlink.org _________________________________ People. Tools. Change. Community. __________________________________________________________________ ___ __ _ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Robert Garcia BigHead Technology 2781 N Carlmont Pl Simi Valley, CA 93065 Phone 805.522.8577 http://www.bighead.net/ http://www.theradmac.com/ [EMAIL PROTECTED] ___________________________________________________________________ ___ __ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Jeff Bohmer VisionLink, Inc. _________________________________ 303.402.0170 www.visionlink.org _________________________________ People. Tools. Change. Community. ____________________________________________________________________ ___ _ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Robert Garcia BigHead Technology 2781 N Carlmont Pl Simi Valley, CA 93065 Phone 805.522.8577 http://www.bighead.net/ http://www.theradmac.com/ [EMAIL PROTECTED] _____________________________________________________________________ ___ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-------------------------- bohmer Mobility Productions http://mobility303.com phone: 303.516.9544 -------------------------- ______________________________________________________________________ __ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body-- Robert Garcia BigHead Technology 2781 N Carlmont Pl Simi Valley, CA 93065 Phone 805.522.8577 http://www.bighead.net/ http://www.theradmac.com/ [EMAIL PROTECTED] _______________________________________________________________________ _ TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED] with unsubscribe witango-talk in the message body
____________________________________________________________________
Jason Pamental, President [EMAIL PROTECTED]
Bathysphere Digital Media Services, Inc. http://bathyspheredms.com
____________________________________________________________________
Tel: 401.490.6830 Fax: 401.490.6831
________________________________________
A North American Distributor for Witango (http://www.witango.us)
Rapid Web Application Development - XML Execution Engine
________________________________________________________________________
TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED]
with unsubscribe witango-talk in the message body
--
________________________________________________________________________
TO UNSUBSCRIBE: send a plain text/US ASCII email to [EMAIL PROTECTED]
with unsubscribe witango-talk in the message body
