We are not able to isolate the issue and was wondering if anyone on the list could give some guidance.
Our environment: Red Hat running ARS 7.5 patch 8 (8 in server group) Oracle 10g running ITSM 7.6 Problem: SRM Events causing all prod servers to crash at once. We see this in the error log: Tue Oct 22 10:14:10 2013 390620 : AR System server terminated when a signal/exception was received by the server (ARNOTE 20) Tue Oct 22 10:14:10 2013 11 Timestamp: 1382451250.0791 Thread Id: 1098680640 Version: 7.5.00 Patch 008 201112090209 Dec 9 2011 02:44:08 ServerName: <servername goes here> Database: SQL -- Oracle Hardware: x86_64 OS: Linux 2.6.18-308.24.1.el5 RPC Id: 442 RPC Call: 3 (CE) RPC Queue: 390620 Client: User Remedy Application Service from Unidentified Client (protocol 14) at IP address <Ip address goes here> Form: WOI:WorkOrderInterface_Create Logging On: Tue Oct 22 10:14:10 2013 390620 : AR System server terminated when a signal/exception was received by the server (ARNOTE 20) Tue Oct 22 10:14:10 2013 Stacks: 0: 0x7e06cd ./arserverd [0x7e06cd] 1: 0x7e0de6 ./arserverd(SignalTrapProc+0x15e) [0x7e0de6] 2: 0x2b81db470be0 /lib64/libpthread.so.0 [0x2b81db470be0] 3: 0x75381a ./arserverd [0x75381a] 4: 0x751a90 ./arserverd [0x751a90] 5: 0x5f84c8 ./arserverd [0x5f84c8] 6: 0x5411ce ./arserverd [0x5411ce] 7: 0x540b8e ./arserverd(arcreateentry_14_svc+0xb0) [0x540b8e] 8: 0x466a41 ./arserverd [0x466a41] 9: 0x7eb0a6 ./arserverd(WorkerThread+0x756) [0x7eb0a6] 10: 0x9e1fb0 ./arserverd(RestartableThreadMain+0x50) [0x9e1fb0] 11: 0x9e1f3b ./arserverd(UnixThreadStartRoutine+0xce) [0x9e1f3b] 12: 0x2b81db46877d /lib64/libpthread.so.0 [0x2b81db46877d] 13: 0x2b81ddf523ed /lib64/libc.so.6(clone+0x6d) [0x2b81ddf523ed] We opened a ticket with BMC and they said that the "CAI Events" plugin was not thread safe. To resolve this we changed the following parameter: Plugin-Filter-API-Threads: 2 2 To just uncommenting it (and accepting the default value): #Plugin-Filter-API-Threads: 1 1 Could it be possible that the other plugins are not thread safe and that they could be contributing to the crash? We still have the following settings in the ar.conf file: Plugin-ARDBC-Threads: 2 2 Plugin-ARDBC-Threads: 2 2 Anyone run into a similar issue before and how did you remediate? Unfortunately upgrading to 7.6 or beyond is not possible at the moment for us ... to many customizations/loss of user tool, etc. Thanks, Sean _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
