Final update. I used the rrrChive product from Real Refined Resources Scandinavia AB (www.rrr.se<http://www.rrr.se>) to export data from every form where the Modified Date > 6/18/2011 The DBA restored the database to 6/19/2011. I reran rrrchive to copy the exported data back in. There was one form that had changed between 6/19/2011 and yesterday so the character field was missing in the old version. After I re-added the field I took the .arx and .idx files for that form and isolated them, ran rrrchive and the data has all been restored.
Before the DBA restored the ARSystem db I exported all definitions with a date greater than 6/18/2011 so I can send them to BMC. Maybe they can find a misplaced "/" in a filter qualification's raw form. Anyway my week in hell is done. The system is happily bouncing between 1% and 60%. Short duration spikes and no cpu or memory creep. Misi, What a great tool. Thanks for supplying something so valuable for such a great price. Thanks to everyone else for the suggestions. You really do learn a lot when the system breaks. --- John J. Reiser Remedy Developer/Administrator Senior Software Development Analyst Lockheed Martin - MS2 The star that burns twice as bright burns half as long. Pay close attention and be illuminated by its brilliance. - paraphrased by me From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: Tuesday, June 28, 2011 1:50 PM To: arslist@ARSLIST.ORG Subject: Re: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) ** DO you mean is the Tomcat executable hogging cpu time? Only for a few minutes after a Tomcat restart. It's the arserver.exe that is eating up cpu cycles. --- John J. Reiser Remedy Developer/Administrator Senior Software Development Analyst Lockheed Martin - MS2 The star that burns twice as bright burns half as long. Pay close attention and be illuminated by its brilliance. - paraphrased by me From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of john.athe...@schneider-electric.com Sent: Tuesday, June 28, 2011 1:17 PM To: arslist@ARSLIST.ORG Subject: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) ** Is the server cacheing or Mid Tier? _____________________________________________________________________________________ John Atherly | APC by Schneider Electric | Information, Process & Organization (IPO) | Remedy Administrator / Developer Phone: +401-789-5735 ext. 2120 | Fax: +401-789-3710 | Email: john.athe...@apcc.com<mailto:%20john.athe...@apcc.com> | Site: www.apc.com/<http://www.apc.com/> | Address: 132 Fairgrounds Road, West Kingston, RI 02892 USA *** Please consider the environment before printing this e-mail Rick Cook <remedyr...@gmail.com> Sent by: "Action Request System discussion list(ARSList)" <arslist@ARSLIST.ORG> 06/28/2011 08:47 AM Please respond to arslist@ARSLIST.ORG To arslist@ARSLIST.ORG cc Subject Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) ** We saw on ESX 3.5 that adding CPUs actually slowed performance, due to a bug in how the processors were accessed. Dropping to 1 or 2 helped a lot, as did upgrading to v4. Rick On Jun 28, 2011 5:33 AM, "Reiser, John J" <john.j.rei...@lmco.com<mailto:john.j.rei...@lmco.com>> wrote: > Mark, > I've been working with BMC Support since last week. They have had me send > many log files trying to capture an event but nothing has shown up yet. I'll > check into Spotlight. > > Theo, > Already disabled escalations, alerts etc. I even disabled every filter with a > notify action because that was the first bit of workflow that would send > arserver.exe running away. The system still overloads. > > Joe, > I'll ask the VM admins to check the NIC settings. Since the SQL Server is > remote and on a huge SAN that side should be ok. The DBA said he saw no > unusual traffic to the ARSystem db. > > Rick, > There ARE FOUR Lights. Sorry can you tell I'm losing it. > The VM has 2 CPUs configured. > VMware Tools says version 8.3.7 build 341836. > > LJ, > I've been capturing Filter, thread, API, SQL logs and sent them to BMC > Support. They see no long queries or transactions that could be causing the > high cpu usage. I did combine sql and api. I'll add the filter logs and see > what kind of timing I get between the three. > > Thanks for the feedback. > If BMC can't solve it today I think I'm going to use Misi's rrrChive and > export and reload the user data after we rollback the meta(?) data from > before the problem started. > > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by me > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG>] On Behalf Of > Walters, Mark > Sent: Tuesday, June 28, 2011 4:59 AM > To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> > Subject: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > corruption? (Long Post) > > Grab a copy of Spotlight on Windows from www.quest.com<http://www.quest.com/> > and you can use it to view the various threads within the arserverd.exe and > work out which one is causing the high CPU load. Once you have this you can > reference the thread/sql/api/filter logs to see what activity it is. > > Mark > > I work for BMC, I don't speak for them. > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG>] On Behalf Of Reiser, > John J > Sent: 27 June 2011 22:27 > To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> > Subject: arserver.exe is consuming 100% cpu - possible DB corruption? (Long > Post) > > Hello Listers, > ARS 7.6.03 > MS 2003 Enterprise > MS SQL 2005 (remote) > Total home grown system. No OOTB modules. > > > I have a real stumper here. It even has BMC scratching their heads. > I have a production system that is experiencing cpu overload that runs up to > 99 in the processes and sits there. > The ARSystem server is virtual machine. We thought maybe it was a MS "Patch > Tuesday" issue and we removed the 10 recent MS patches one at a time and > restarted the machine each time. The problem still exists after the arserver > service starts. Sometime immediately and sometimes it will sit for 1- 20 > minutes before it starts to hog the CPUs. > To eliminate any other OS and file system issues we grabbed a two week old > backup image of the server and restored it. > The system came back ok for a short while and then started to lock up the CPU > again. > Working with BMC I set the logs on and restarted. We saw the system jump to > 100% within a minute and captured a 10MB arsql.log file. > It can force the overload at anytime by firing filter workflow with a > notification action in it. > I disabled this one filter but the system still loaded up. I added a Filter > that ran a 0 and the only action was Goto 1000 to jump all Filter actions > that fired on the change of the Status field in question. > Still no joy. > I've disabled every piece of Notify workflow. That worked the best and kept > the system alive for longer stretches but we can't run a system that way. > > I've come to the realization that there may be corrupted information in the > DB object tables and I wanted to get some feedback. > Using rrrChive I can pull a copy of every form's data since, say, two weeks > ago. Then have the DBA restore the entire system from that date. After the > restore I would use rrrChive to reload the two weeks' data (Modified date' > > "06/11/2011") and hope for the best. > > Any workflow that was changed in the last two weeks is negligible and could > be recreated/updated as needed. > > Do you think this is a viable solution? > When I asked the BMC tech if I could dump the T,H & B tables ; restore the db > and reload the T, H & B tables he reminded me that the arschema and other > meta tables would probably be out of synch. > That's when I thought of using rrrChive. > > Sorry to be so long winded but I need to get this back online, BMC can't find > anything in the logs and I don't want to lose the tickets we've taken in the > last week. > > > > > --- > John J. Reiser > Remedy Developer/Administrator > Senior Software Development Analyst > Lockheed Martin - MS2 > The star that burns twice as bright burns half as long. > Pay close attention and be illuminated by its brilliance. - paraphrased by me > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at > www.arslist.org<http://www.arslist.org/> attend wwrug11 > www.wwrug.com<http://www.wwrug.com/> ARSList: "Where the Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at > www.arslist.org<http://www.arslist.org/> > attend wwrug11 www.wwrug.com<http://www.wwrug.com/> ARSList: "Where the > Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at > www.arslist.org<http://www.arslist.org/> > attend wwrug11 www.wwrug.com<http://www.wwrug.com/> ARSList: "Where the > Answers Are" _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
