Recon jobs, should really only have to run weekly.. but good point.. purge jobs nightly..
On Tue, Jun 28, 2011 at 11:41 AM, dcharters < dchart...@www.charterssoftware.com> wrote: > Do you have AIE and/or Recon jobs running at all? Are those engines even > installed anywhere? > I have also seen this with special characters being passed in Diary fields. > That was a few versions back but something to look at. > > > On Tue, 28 Jun 2011 08:30:24 -0400, Reiser, John J wrote > > Mark, > > I've been working with BMC Support since last week. They have had > > me send many log files trying to capture an event but nothing has > > shown up yet. I'll check into Spotlight. > > > > Theo, > > Already disabled escalations, alerts etc. I even disabled every > > filter with a notify action because that was the first bit of > > workflow that would send arserver.exe running away. The system still > > overloads. > > > > Joe, > > I'll ask the VM admins to check the NIC settings. Since the SQL > > Server is remote and on a huge SAN that side should be ok. The DBA > > said he saw no unusual traffic to the ARSystem db. > > > > Rick, > > There ARE FOUR Lights. Sorry can you tell I'm losing it. > > The VM has 2 CPUs configured. > > VMware Tools says version 8.3.7 build 341836. > > > > LJ, > > I've been capturing Filter, thread, API, SQL logs and sent them to > > BMC Support. They see no long queries or transactions that could be > > causing the high cpu usage. I did combine sql and api. I'll add the > > filter logs and see what kind of timing I get between the three. > > > > Thanks for the feedback. > > If BMC can't solve it today I think I'm going to use Misi's rrrChive > > and export and reload the user data after we rollback the meta(?) > > data from before the problem started. > > > > --- > > John J. Reiser > > Remedy Developer/Administrator > > Senior Software Development Analyst > > Lockheed Martin - MS2 > > The star that burns twice as bright burns half as long. > > Pay close attention and be illuminated by its brilliance. - > > paraphrased by me > > > > -----Original Message----- > > From: Action Request System discussion list(ARSList) > > [mailto:arslist@ARSLIST.ORG] On Behalf Of Walters, Mark Sent: > > Tuesday, June 28, 2011 4:59 AM To: arslist@ARSLIST.ORG Subject: > > EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB > > corruption? (Long Post) > > > > Grab a copy of Spotlight on Windows from www.quest.com and you can > > use it to view the various threads within the arserverd.exe and work > > out which one is causing the high CPU load. Once you have this you > > can reference the thread/sql/api/filter logs to see what activity it > > is. > > > > Mark > > > > I work for BMC, I don't speak for them. > > > > -----Original Message----- > > From: Action Request System discussion list(ARSList) > > [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: 27 > > June 2011 22:27 To: arslist@ARSLIST.ORG Subject: arserver.exe is > > consuming 100% cpu - possible DB corruption? (Long Post) > > > > Hello Listers, > > ARS 7.6.03 > > MS 2003 Enterprise > > MS SQL 2005 (remote) > > Total home grown system. No OOTB modules. > > > > I have a real stumper here. It even has BMC scratching their heads. > > I have a production system that is experiencing cpu overload that > > runs up to 99 in the processes and sits there. The ARSystem server > > is virtual machine. We thought maybe it was a MS "Patch Tuesday" > > issue and we removed the 10 recent MS patches one at a time and > > restarted the machine each time. The problem still exists after the > > arserver service starts. Sometime immediately and sometimes it will > > sit for 1- 20 minutes before it starts to hog the CPUs. To eliminate > > any other OS and file system issues we grabbed a two week old backup > > image of the server and restored it. The system came back ok for a > > short while and then started to lock up the CPU again. Working with > > BMC I set the logs on and restarted. We saw the system jump to 100% > > within a minute and captured a 10MB arsql.log file. It can force the > > overload at anytime by firing filter workflow with a notification > > action in it. I disabled this one filter but the system still loaded > > up. I added a Filter that ran a 0 and the only action was Goto 1000 > > to jump all Filter actions that fired on the change of the Status > > field in question. Still no joy. I've disabled every piece of Notify > > workflow. That worked the best and kept the system alive for longer > > stretches but we can't run a system that way. > > > > I've come to the realization that there may be corrupted information > > in the DB object tables and I wanted to get some feedback. Using > > rrrChive I can pull a copy of every form's data since, say, two > > weeks ago. Then have the DBA restore the entire system from that > > date. After the restore I would use rrrChive to reload the two > > weeks' data (Modified date' > "06/11/2011") and hope for the best. > > > > Any workflow that was changed in the last two weeks is negligible > > and could be recreated/updated as needed. > > > > Do you think this is a viable solution? > > When I asked the BMC tech if I could dump the T,H & B tables ; > > restore the db and reload the T, H & B tables he reminded me that > > the arschema and other meta tables would probably be out of synch. > > That's when I thought of using rrrChive. > > > > Sorry to be so long winded but I need to get this back online, BMC > > can't find anything in the logs and I don't want to lose the tickets > > we've taken in the last week. > > > > --- > > John J. Reiser > > Remedy Developer/Administrator > > Senior Software Development Analyst > > Lockheed Martin - MS2 > > The star that burns twice as bright burns half as long. > > Pay close attention and be illuminated by its brilliance. - > > paraphrased by me > > > > > > _____________________________________________________________________________ > __ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend > > wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > > > > _____________________________________________________________________________ > __ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > > > > _____________________________________________________________________________ > __ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > -- > Open WebMail Project (http://openwebmail.org) > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > -- Patrick Zandi _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
