Mark, I've been working with BMC Support since last week. They have had me send many log files trying to capture an event but nothing has shown up yet. I'll check into Spotlight.
Theo, Already disabled escalations, alerts etc. I even disabled every filter with a notify action because that was the first bit of workflow that would send arserver.exe running away. The system still overloads. Joe, I'll ask the VM admins to check the NIC settings. Since the SQL Server is remote and on a huge SAN that side should be ok. The DBA said he saw no unusual traffic to the ARSystem db. Rick, There ARE FOUR Lights. Sorry can you tell I'm losing it. The VM has 2 CPUs configured. VMware Tools says version 8.3.7 build 341836. LJ, I've been capturing Filter, thread, API, SQL logs and sent them to BMC Support. They see no long queries or transactions that could be causing the high cpu usage. I did combine sql and api. I'll add the filter logs and see what kind of timing I get between the three. Thanks for the feedback. If BMC can't solve it today I think I'm going to use Misi's rrrChive and export and reload the user data after we rollback the meta(?) data from before the problem started. --- John J. Reiser Remedy Developer/Administrator Senior Software Development Analyst Lockheed Martin - MS2 The star that burns twice as bright burns half as long. Pay close attention and be illuminated by its brilliance. - paraphrased by me -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Walters, Mark Sent: Tuesday, June 28, 2011 4:59 AM To: arslist@ARSLIST.ORG Subject: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) Grab a copy of Spotlight on Windows from www.quest.com and you can use it to view the various threads within the arserverd.exe and work out which one is causing the high CPU load. Once you have this you can reference the thread/sql/api/filter logs to see what activity it is. Mark I work for BMC, I don't speak for them. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: 27 June 2011 22:27 To: arslist@ARSLIST.ORG Subject: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) Hello Listers, ARS 7.6.03 MS 2003 Enterprise MS SQL 2005 (remote) Total home grown system. No OOTB modules. I have a real stumper here. It even has BMC scratching their heads. I have a production system that is experiencing cpu overload that runs up to 99 in the processes and sits there. The ARSystem server is virtual machine. We thought maybe it was a MS "Patch Tuesday" issue and we removed the 10 recent MS patches one at a time and restarted the machine each time. The problem still exists after the arserver service starts. Sometime immediately and sometimes it will sit for 1- 20 minutes before it starts to hog the CPUs. To eliminate any other OS and file system issues we grabbed a two week old backup image of the server and restored it. The system came back ok for a short while and then started to lock up the CPU again. Working with BMC I set the logs on and restarted. We saw the system jump to 100% within a minute and captured a 10MB arsql.log file. It can force the overload at anytime by firing filter workflow with a notification action in it. I disabled this one filter but the system still loaded up. I added a Filter that ran a 0 and the only action was Goto 1000 to jump all Filter actions that fired on the change of the Status field in question. Still no joy. I've disabled every piece of Notify workflow. That worked the best and kept the system alive for longer stretches but we can't run a system that way. I've come to the realization that there may be corrupted information in the DB object tables and I wanted to get some feedback. Using rrrChive I can pull a copy of every form's data since, say, two weeks ago. Then have the DBA restore the entire system from that date. After the restore I would use rrrChive to reload the two weeks' data (Modified date' > "06/11/2011") and hope for the best. Any workflow that was changed in the last two weeks is negligible and could be recreated/updated as needed. Do you think this is a viable solution? When I asked the BMC tech if I could dump the T,H & B tables ; restore the db and reload the T, H & B tables he reminded me that the arschema and other meta tables would probably be out of synch. That's when I thought of using rrrChive. Sorry to be so long winded but I need to get this back online, BMC can't find anything in the logs and I don't want to lose the tickets we've taken in the last week. --- John J. Reiser Remedy Developer/Administrator Senior Software Development Analyst Lockheed Martin - MS2 The star that burns twice as bright burns half as long. Pay close attention and be illuminated by its brilliance. - paraphrased by me _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
