We are in the same boat, but the cause of the crash is different. Everything was happy on patch 1, only after installing patch 2 did these start.
In my stack traces, the following information is very informative: RPC Id: 28155 RPC Call: 3 (CE) RPC Queue: 390620 Client: User xxxxxx from Remedy User (protocol 11) at IP address 1.2.3.4 I can tell that a create entry caused the sig11, and also which user. By grepping the filter, sql, and api logs for the rpc id, I can see what that user was doing. Problem is that I can't reproduce the issue on our development environment when triggering the same workflow. It is intermittent at this point, as even the user that causes the problem has most of the same operations complete successfully. Still something I'm not seeing. Axton Grams On 5/2/07, Ben Cantatore <[EMAIL PROTECTED]> wrote:
** Axton, I'm not entirely sure if I had the same problem or not, but I was getting those on patch 1. I cleaned up a number of issues with my enviroment (mostly bad workflow) so its hard to say what actually fixed the problem in my case. I've been running stable for 2 months now. I stumbled upon a user action that would recreate the problem. That specific user action was fixed by applying patch 3 CMDB/Asset. I suspected that other user actions might have been triggering this problem as well, but was never able to figure out what else. When I had this problem it would cause a 3 minute outage and then recover on its own. My configuration is Linux Suse 9 connecting to an Oracle 10g db. I have patch 2 server, patch 3 ITSM on my dev enviroment and haven't seen this problem so far. Fri Mar 30 11:18:58 2007 390600 : AR System server terminated -- fatal error encountered (ARNOTE 21) Fri Mar 30 11:18:58 2007 390600 : AR System server terminated when a signal/exception was received by the server (ARNOTE 20) Fri Mar 30 11:18:58 2007 11 Timestamp: Fri Mar 30 2007 11:18:58.4915 Thread Id: 1139297200 Version: 7.0.01 patch 001 200701111654 Jan 11 2007 17:27:28 ServerName: ryelxpitsmapp1 Database: SQL -- Oracle Hardware: i686 OS: Linux 2.6.5-7.244-bigsmp RPC Id: 0 RPC Call: 0 RPC Queue: 390600 Protocol Version: 12 Client IP Address: Logging On: Fri Mar 30 11:18:58 2007 390600 : AR System server terminated when a signal/exception was received by the server (ARNOTE 20) Fri Mar 30 11:18:58 2007 Stacks: 0: 83153BB ./arserverd [0x83153bb] 1: 8315AFD ./arserverd [0x8315afd] 2: FFFFE420 [0xffffe420] 3: 44021E19 /usr/arsystem/ryelxpitsmapp1/cmdb/server/bin/libarxmlutil.so [0x44021e19] 4: 421380E0 /lib/tls/libc.so.6(exit+0x60) [0x421380e0] 5: 82A675A ./arserverd [0x82a675a] 6: 82A67B7 ./arserverd [0x82a67b7] 7: 834FCBF ./arserverd [0x834fcbf] 8: 8238011 ./arserverd [0x8238011] 9: 831BB0C ./arserverd [0x831bb0c] 10: 84C28E3 ./arserverd [0x84c28e3] 11: 84C2884 ./arserverd [0x84c2884] 12: 40DC3CF7 /lib/tls/libpthread.so.0 [0x40dc3cf7] 13: 421C821E /lib/tls/libc.so.6(__clone+0x5e) [0x421c821e] Fri Mar 30 11:19:01 2007 390600 : Failure while trying to connect to the SQL database. Please ensure the SQL database is running or contact the Database Administrator for help (ARERR 550) Fri Mar 30 11:19:01 2007 ORA-12545: Connect failed because target host or object does not exist Fri Mar 30 11:19:02 2007 390600 : Cannot initialize contact with SQL database (ARERR 551) Fri Mar 30 11:19:02 2007 Stop server Ben Cantatore Remedy Administrator Avon (914) 935-2946 *Axton <[EMAIL PROTECTED]>* Sent by: "Action Request System discussion list(ARSList)" < [email protected]> 05/01/2007 03:29 PM Please respond to [email protected] To [email protected] cc Subject 7.0.1 Patch2 Solaris - Problems Has anyone upgraded to 7.0.1 patch2 on Solaris? Our server is bouncing pretty often after upgrading to patch 2. Is anyone else experiencing this? We get the following stack trace in the arerror.log file each time the server bounces: Tue May 1 13:33:03 2007 390620 : AR System server terminated when a signal/exception was received by the server (ARNOTE 20) Tue May 1 13:33:03 2007 11 Timestamp: Tue May 01 2007 13:33:03.5130 Thread Id: 33 Version: 7.0.01 patch 002 200704021644 Apr 2 2007 17:18:55 ServerName: rm00p10 Database: SQL -- Oracle Hardware: sun4u OS: SunOS 5.9 RPC Id: 28155 RPC Call: 3 (CE) RPC Queue: 390620 Client: User xxxxxx from Remedy User (protocol 11) at IP address 1.2.3.4 Form: Logging On: API Filter SQL User Thread Stacks: /prod/sys/remedy/bin/arserverd:DumpStackTrace+0x88 /prod/sys/remedy/bin/arserverd:SignalTrapProc+0x160 /usr/lib/libthread.so.1:0x15bac /usr/lib/libthread.so.1:0xf804 /usr/lib/libthread.so.1:0xf9b4 /usr/platform/sun4u-us3/lib/libc_psr.so.1:memset+0x140 [ Signal 11 (SEGV)] /prod/sys/remedy/bin/arserverd:SendNotification+0x41fc The stack traces vary slightly: Stacks: /prod/sys/remedy/bin/arserverd:DumpStackTrace+0x88 /prod/sys/remedy/bin/arserverd:SignalTrapProc+0x160 /usr/lib/libthread.so.1:0x15bac /usr/lib/libthread.so.1:0xf804 /usr/lib/libthread.so.1:0xf9b4 /usr/platform/sun4u-us3/lib/libc_psr.so.1:memset+0x140 [ Signal 11 (SEGV)] /prod/sys/remedy/bin/arserverd:SendNotification+0x41fc /prod/sys/remedy/bin/arserverd:DumpStackTrace+0x88 /prod/sys/remedy/bin/arserverd:SignalTrapProc+0x160 /usr/lib/libthread.so.1:0x15bac /usr/lib/libthread.so.1:0xf804 /usr/lib/libthread.so.1:0xf9b4 /usr/platform/sun4u-us3/lib/libc_psr.so.1:memset+0x140 [ Signal 11 (SEGV)] /prod/sys/remedy/bin/arserverd:SendNotification+0x41fc /prod/sys/remedy/bin/arserverd:DumpStackTrace+0x88 /prod/sys/remedy/bin/arserverd:SignalTrapProc+0x160 /usr/lib/libthread.so.1:0x15bac /usr/lib/libthread.so.1:0xf804 /usr/lib/libthread.so.1:0xf9b4 /usr/lib/libc.so.1:strlen+0x1c [ Signal 11 (SEGV)] /prod/sys/remedy/bin/arserverd:Ar_xdr_string+0xf0 /prod/sys/remedy/bin/arserverd:xdr_ArCharLimitsStruct+0x38 /prod/sys/remedy/bin/arserverd:xdr_ArFieldLimitStruct+0x9c /usr/lib/libnsl.so.1:xdr_free+0x14 /prod/sys/remedy/bin/arserverd:V8ToV1FieldLimitList+0x13c /prod/sys/remedy/bin/arserverd:argetmultiplefields_7_svc+0x760 /prod/sys/remedy/bin/arserverd:HandleRPCs+0x134e0 /prod/sys/remedy/bin/arserverd:WorkerThread+0x5f4 /prod/sys/remedy/bin/arserverd:RestartableThreadMain+0x60 /prod/sys/remedy/bin/arserverd:UnixThreadStartRoutine+0xe8 /usr/lib/libthread.so.1:0x15854 /prod/sys/remedy/bin/arserverd:DumpStackTrace+0x88 /prod/sys/remedy/bin/arserverd:SignalTrapProc+0x160 /usr/lib/libthread.so.1:0x15bac /usr/lib/libthread.so.1:0xf804 /usr/lib/libthread.so.1:0xf9b4 /usr/platform/sun4u-us3/lib/libc_psr.so.1:memset+0x140 [ Signal 11 (SEGV)] /prod/sys/remedy/bin/arserverd:SendNotification+0x41fc Thanks, Axton Grams _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are" __20060125_______________________This posting was submitted with HTML in it___
_______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
