OK. So, it ain't a denied malloc. You got a bug that somehow you are exposing and seemingly no one else is. Tough one. BMC could give you a debug build, you'd generate a core, and give it to some BMC folks. That would require some very high level of support I would assume (though I have seen that ages ago).
I still think the best bet is to trace and hunt around filters to see if anything unusual is the culprit. As Mark said, make sure your system is taking cores. You have a process ending with SIGSEGV; that always ends up with a core. I've never seen a system where it didn't but then I am no Unix admin. There are other options such as plugging in stuff to log system status at intervals etc. But your best bet is definitely to turn on logging and review logs. Try to reproduce this in a safe (non-production) environment. BTW, the OS or armonitor doesn't terminate the process. This is a hardware interrupt that the OS catches. ARS will have a signal handler and that gets control. That is what prints the Signal 11 trace. Are you running ITSM? You didn't say. But code paths within the arserver will vary even with the same ARS applications for so many reasons as to make it astronomical. Were there any filter / db / environment changes just before the symptoms started showing up? These are good areas to investigate as well. These types of problems can be solved (circumvented) quickly or can take forever (ie not get solved). But they are quite involved and interesting. Good luck! Ben From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of patrick zandi Sent: September-23-11 15:38 To: [email protected] Subject: Re: ARS 7.1 P6 Server -- 4 days restarting (possible memory OS 32bit issue) signal is 11 ** Perplexing is the (no core bombs) We are running fine and them Boom, we get that signal: and armonitor shuts the arserverd down and restarts a new one.. What I was hoping was someone would Say I remember that.. and Patch X,Y,Z fixed that.. I did conplete logging and the log stops and the log starts (nothing after the signal and startup).. Clear.. Clean.. I looked at all the patches and fine the following:: but cannot put a finger on any one of them (Specifically) (NL--not likely) (ML- most likely) SW00351599 The AR System server crashed when a filter used converted values in a Set Field action. SW00351922 The AR System server program terminated when building a userList to send notifications. ML-SW00356647 Too many filters executed recursively causing a stack overflow, which resulted in failure of the AR System server. SW00328336 The AR System server crashed while saving text to a Diary field on which audit was enabled. SW00328337 arserverd crashed while attempting to communicate with the plug-in server through PluginServerCallWithRetry. SW00337127 A memory leak issue occurred in the AR System server. NL-SW00338411 The AR System server crashed during an archiving process if the archive source form contained non-data fields such as Text, Trim, Button, and so on. NL-SW00346370 The AR System server crashed while processing an error handling filter. SW00314816 When a user performed a search on a Join form, but did not have permissions to view all records returned in the result, the AR System server crashed. NL--SW00322802 Creating an entry with a user name larger than 180 bytes caused the AR System server to crash when the status history was being recorded and the initial status was not New. On Fri, Sep 23, 2011 at 5:09 AM, Ben Chernys <[email protected]> wrote: ** Hi Mark, Patrick, Signal 11 is SIGSEGV which is not necessarily a malloc failure though indeed a malloc failure may lead to it. It is not always possible to log malloc failures - after all it takes some memory to cut a log record. A segmentation violation is always the result of bad code (accessing memory not allocated to the process or not in the processes address space - which 0 is a candidate (malloc's return value on failure)). That being said, it is possible to not trigger the execution path with that bad code by altering filters etc, so definitely the route to go on is along the lines that Mark talked: the core is always a wealth of info - even though ARS will not have debugging compiled in ;-) I would also turn on all logging. SQL, API, Filter on the server, and unlimited, and pointing to the same file until the next occurrence. Then you will have a wealth of ARS information to go through. Generally something will stand out. Recursive filter loops are usually trapped by the maximum filter limit - though if that is set high enough the process will run out of memory before hitting up against that. If yours is high, you could try setting it lower. You may also want to go to a higher patch level if one is available. I am no longer that familiar with the patches available on 7.1. Also, I know that memory on Solaris may be restricted by the admin. (I forget the commands to determine this - but they will be easily found on the web). ulimits Perhaps? Cheers Ben Chernys Senior Software Architect Software Tool House Inc. Canada / Deutschland / Germany Mobile: +49 171 380 2329 <tel:%2B49%20171%20380%202329> GMT + 1 + [ DST ] Email: <mailto:[email protected]> Ben.Chernys _AT_ softwaretoolhouse.com Web: <http://www.softwaretoolhouse.com> www.softwaretoolhouse.com Check out Software Tool House's free Diary Editor. Meta-Update, our premium ARS Data tool, lets you automate your imports, migrations, in no time at all, without programming, without staging forms, without merge workflow. <http://www.softwaretoolhouse.com/> http://www.softwaretoolhouse.com/ From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of Walters, Mark Sent: September-23-11 09:08 To: [email protected] Subject: Re: ARS 7.1 P6 Server -- 4 days restarting (possible memory OS 32bit issue) signal is 11 ** It may be memory but I would expect to see malloc errors (ARERR 300) in the arerror.log if this was the case. The fact you're not seeing a stack trace like this; Mon Sep 20 08:33:52 2010 6 Timestamp: Mon Sep 20 2010 08:33:52.1865 Thread Id: 4 Version: 7.1.00 Patch 009 201009200800 ServerName: test71 Database: SQL -- Oracle Hardware: sun4u OS: SunOS 5.10 RPC Id: 337 RPC Call: 106 (GLXS) RPC Queue: 390600 Client: User Demo from Remedy Administrator (protocol 13) at IP address 192.168.1.54 Form: Logging On: suggests it may be a recursive filter - on Solaris this often causes a crash without logging anything useful. Check to see whether there are any core files in the server/bin directory as this is another symptom of this type of crash on Solaris. If cores are enabled (check with the OS coreadm command) then the server may create them even though you're not running a debug build. If you do have some core files then run the pstack command against them (pstack core) and you will be able to see the stack of each thread within the server - if it is a recursive filter causing a stack overflow then one of the threads should stand out as being much bigger than the others. Depending on what you see you may then need to enable FILTER/SQL logging to try and capture the workflow that is causing the crash. It's also worth checking the Filter-Max-Stack value in ar.conf - various installers set this to a very high value - try reducing it back down to 50 or so and this should stop most filter recursion crashes and log an error instead. Mark I work for BMC, I don't speak for them. From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of patrick zandi Sent: 22 September 2011 21:07 To: [email protected] Subject: ARS 7.1 P6 Server -- 4 days restarting (possible memory OS 32bit issue) signal is 11 ** Just a Quick Question:: ARS 7.1 P6 :: on solaris 10, I am seeing a Operating system telling the ars to shutdown about every 4 -6 days.. not positive, nothing in debugging of logs at all, only in the ARMONITOR.log where it says.. 2011 ARMonitor child process (pid:15277) died with 11. And the signal is 11. ./arserverd Can I assume Signal 11 is Memory? --- I have seen alot of memory issues with a 11 signal in the arslist... -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
