SLM -- if you are running SLM it is a major Pig Also shut off the apache if you can
On Tue, Jun 28, 2011 at 8:30 AM, Reiser, John J <john.j.rei...@lmco.com>wrote: > Mark, > I've been working with BMC Support since last week. They have had me send > many log files trying to capture an event but nothing has shown up yet. I'll > check into Spotlight. > > Theo, > Already disabled escalations, alerts etc. I even disabled every filter > with a notify action because that was the first bit of workflow that would > send arserver.exe running away. The system still overloads. > > Joe, > I'll ask the VM admins to check the NIC settings. Since the SQL Server is > remote and on a huge SAN that side should be ok. The DBA said he saw no > unusual traffic to the ARSystem db. > > Rick, > There ARE FOUR Lights. Sorry can you tell I'm losing it. > The VM has 2 CPUs configured. > VMware Tools says version 8.3.7 build 341836. > > LJ, > I've been capturing Filter, thread, API, SQL logs and sent them to BMC > Support. They see no long queries or transactions that could be causing the > high cpu usage. I did combine sql and api. I'll add the filter logs and see > what kind of timing I get between the three. > > Thanks for the feedback. > If BMC can't solve it today I think I'm going to use Misi's rrrChive and > export and reload the user data after we rollback the meta(?) data from > before the problem started. > > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by > me > > -----Original Message----- > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] On Behalf Of Walters, Mark > Sent: Tuesday, June 28, 2011 4:59 AM > To: arslist@ARSLIST.ORG > Subject: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > corruption? (Long Post) > > Grab a copy of Spotlight on Windows from www.quest.com and you can use it > to view the various threads within the arserverd.exe and work out which one > is causing the high CPU load. Once you have this you can reference the > thread/sql/api/filter logs to see what activity it is. > > Mark > > I work for BMC, I don't speak for them. > > -----Original Message----- > From: Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] On Behalf Of Reiser, John J > Sent: 27 June 2011 22:27 > To: arslist@ARSLIST.ORG > Subject: arserver.exe is consuming 100% cpu - possible DB corruption? (Long > Post) > > Hello Listers, > ARS 7.6.03 > MS 2003 Enterprise > MS SQL 2005 (remote) > Total home grown system. No OOTB modules. > > > I have a real stumper here. It even has BMC scratching their heads. > I have a production system that is experiencing cpu overload that runs up > to 99 in the processes and sits there. > The ARSystem server is virtual machine. We thought maybe it was a MS "Patch > Tuesday" issue and we removed the 10 recent MS patches one at a time and > restarted the machine each time. The problem still exists after the arserver > service starts. Sometime immediately and sometimes it will sit for 1- 20 > minutes before it starts to hog the CPUs. > To eliminate any other OS and file system issues we grabbed a two week old > backup image of the server and restored it. > The system came back ok for a short while and then started to lock up the > CPU again. > Working with BMC I set the logs on and restarted. We saw the system jump to > 100% within a minute and captured a 10MB arsql.log file. > It can force the overload at anytime by firing filter workflow with a > notification action in it. > I disabled this one filter but the system still loaded up. I added a Filter > that ran a 0 and the only action was Goto 1000 to jump all Filter actions > that fired on the change of the Status field in question. > Still no joy. > I've disabled every piece of Notify workflow. That worked the best and kept > the system alive for longer stretches but we can't run a system that way. > > I've come to the realization that there may be corrupted information in the > DB object tables and I wanted to get some feedback. > Using rrrChive I can pull a copy of every form's data since, say, two weeks > ago. Then have the DBA restore the entire system from that date. After the > restore I would use rrrChive to reload the two weeks' data (Modified date' > > "06/11/2011") and hope for the best. > > Any workflow that was changed in the last two weeks is negligible and could > be recreated/updated as needed. > > Do you think this is a viable solution? > When I asked the BMC tech if I could dump the T,H & B tables ; restore the > db and reload the T, H & B tables he reminded me that the arschema and other > meta tables would probably be out of synch. > That's when I thought of using rrrChive. > > Sorry to be so long winded but I need to get this back online, BMC can't > find anything in the logs and I don't want to lose the tickets we've taken > in the last week. > > > > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by > me > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 > www.wwrug.com ARSList: "Where the Answers Are" > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > -- Patrick Zandi _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
