Are you sure there is not a user out there performing an unqualified search or full text search? I assume you deny unqualified searching and limit the returns for a get list?
Regards, Andrew Goodall Software Engineer 2 | Development Services | jcpenney . www.jcp.com <http://www.jcp.com/> ________________________________ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: Tuesday, June 28, 2011 10:38 AM To: arslist@ARSLIST.ORG Subject: Re: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) Rick, I've gone in a few times and set the affinity to CPU0. I know that's not exactly what you mean but it allows me to do other things like look at log files because the arserver.exe is only hitting one of the cpus. I'll check on the RunIf qualifications but not a whole lot of workflow was touched recently. The problem just sort of appeared. Though I was in Dev Studio at the time someone reported the outage to me I wasn't actually working in Dev Studio. When I came back to the window it was hung. So was the Usertool on my PC. We don't have any OOTB modules with the exception of maybe the Samples. No ITSM, No CMDB. This is a basic ARServer system with all home built forms. And nothing new has been added that didn't get built on the development VM. And that one works fine. We even stopped both ARservers, pointed the Dev server to the Production database and the problem came with it. That's what makes me think there is something wrong in the part of the db that houses the definitions. Restoring the db to a state prior to the hiccup on Tuesday should eliminate any object changes. Then I can re-import all of the data records from the forms that I exported before the restore. If we restore and cannot reproduce the error then re-import and it comes back then it could be the Diary Field issue but I hope that doesn't happen. During this past week I've disabled notification filters, stopped the email engine, stopped Tomcat. The problem comes back and BMC doesn't see anything in the logs. --- John J. Reiser Remedy Developer/Administrator Senior Software Development Analyst Lockheed Martin - MS2 The star that burns twice as bright burns half as long. Pay close attention and be illuminated by its brilliance. - paraphrased by me From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Rick Cook Sent: Tuesday, June 28, 2011 10:50 AM To: arslist@ARSLIST.ORG Subject: EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB corruption? (Long Post) ** There is also a hotfix for 7.1 involving some embedded spaces in Filter Run If quals that was causing similar problems. Wouldn't put it past the devs to reintroduce a problem that was fixed in a previous version. In fact, I just fixed one of those yesterday. Rick On Jun 28, 2011 7:46 AM, "dcharters" <dchart...@www.charterssoftware.com> wrote: > Do you have AIE and/or Recon jobs running at all? Are those engines even > installed anywhere? > I have also seen this with special characters being passed in Diary fields. > That was a few versions back but something to look at. > > > On Tue, 28 Jun 2011 08:30:24 -0400, Reiser, John J wrote >> Mark, >> I've been working with BMC Support since last week. They have had >> me send many log files trying to capture an event but nothing has >> shown up yet. I'll check into Spotlight. >> >> Theo, >> Already disabled escalations, alerts etc. I even disabled every >> filter with a notify action because that was the first bit of >> workflow that would send arserver.exe running away. The system still >> overloads. >> >> Joe, >> I'll ask the VM admins to check the NIC settings. Since the SQL >> Server is remote and on a huge SAN that side should be ok. The DBA >> said he saw no unusual traffic to the ARSystem db. >> >> Rick, >> There ARE FOUR Lights. Sorry can you tell I'm losing it. >> The VM has 2 CPUs configured. >> VMware Tools says version 8.3.7 build 341836. >> >> LJ, >> I've been capturing Filter, thread, API, SQL logs and sent them to >> BMC Support. They see no long queries or transactions that could be >> causing the high cpu usage. I did combine sql and api. I'll add the >> filter logs and see what kind of timing I get between the three. >> >> Thanks for the feedback. >> If BMC can't solve it today I think I'm going to use Misi's rrrChive >> and export and reload the user data after we rollback the meta(?) >> data from before the problem started. >> >> --- >> John J. Reiser >> Remedy Developer/Administrator >> Senior Software Development Analyst >> Lockheed Martin - MS2 >> The star that burns twice as bright burns half as long. >> Pay close attention and be illuminated by its brilliance. - >> paraphrased by me >> >> -----Original Message----- >> From: Action Request System discussion list(ARSList) >> [mailto:arslist@ARSLIST.ORG] On Behalf Of Walters, Mark Sent: >> Tuesday, June 28, 2011 4:59 AM To: arslist@ARSLIST.ORG Subject: >> EXTERNAL: Re: arserver.exe is consuming 100% cpu - possible DB >> corruption? (Long Post) >> >> Grab a copy of Spotlight on Windows from www.quest.com and you can >> use it to view the various threads within the arserverd.exe and work >> out which one is causing the high CPU load. Once you have this you >> can reference the thread/sql/api/filter logs to see what activity it >> is. >> >> Mark >> >> I work for BMC, I don't speak for them. >> >> -----Original Message----- >> From: Action Request System discussion list(ARSList) >> [mailto:arslist@ARSLIST.ORG] On Behalf Of Reiser, John J Sent: 27 >> June 2011 22:27 To: arslist@ARSLIST.ORG Subject: arserver.exe is >> consuming 100% cpu - possible DB corruption? (Long Post) >> >> Hello Listers, >> ARS 7.6.03 >> MS 2003 Enterprise >> MS SQL 2005 (remote) >> Total home grown system. No OOTB modules. >> >> I have a real stumper here. It even has BMC scratching their heads. >> I have a production system that is experiencing cpu overload that >> runs up to 99 in the processes and sits there. The ARSystem server >> is virtual machine. We thought maybe it was a MS "Patch Tuesday" >> issue and we removed the 10 recent MS patches one at a time and >> restarted the machine each time. The problem still exists after the >> arserver service starts. Sometime immediately and sometimes it will >> sit for 1- 20 minutes before it starts to hog the CPUs. To eliminate >> any other OS and file system issues we grabbed a two week old backup >> image of the server and restored it. The system came back ok for a >> short while and then started to lock up the CPU again. Working with >> BMC I set the logs on and restarted. We saw the system jump to 100% >> within a minute and captured a 10MB arsql.log file. It can force the >> overload at anytime by firing filter workflow with a notification >> action in it. I disabled this one filter but the system still loaded >> up. I added a Filter that ran a 0 and the only action was Goto 1000 >> to jump all Filter actions that fired on the change of the Status >> field in question. Still no joy. I've disabled every piece of Notify >> workflow. That worked the best and kept the system alive for longer >> stretches but we can't run a system that way. >> >> I've come to the realization that there may be corrupted information >> in the DB object tables and I wanted to get some feedback. Using >> rrrChive I can pull a copy of every form's data since, say, two >> weeks ago. Then have the DBA restore the entire system from that >> date. After the restore I would use rrrChive to reload the two >> weeks' data (Modified date' > "06/11/2011") and hope for the best. >> >> Any workflow that was changed in the last two weeks is negligible >> and could be recreated/updated as needed. >> >> Do you think this is a viable solution? >> When I asked the BMC tech if I could dump the T,H & B tables ; >> restore the db and reload the T, H & B tables he reminded me that >> the arschema and other meta tables would probably be out of synch. >> That's when I thought of using rrrChive. >> >> Sorry to be so long winded but I need to get this back online, BMC >> can't find anything in the logs and I don't want to lose the tickets >> we've taken in the last week. >> >> --- >> John J. Reiser >> Remedy Developer/Administrator >> Senior Software Development Analyst >> Lockheed Martin - MS2 >> The star that burns twice as bright burns half as long. >> Pay close attention and be illuminated by its brilliance. - >> paraphrased by me >> >> > ________________________________________________________________________ _____ > __ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend >> wwrug11 www.wwrug.com ARSList: "Where the Answers Are" >> >> > ________________________________________________________________________ _____ > __ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" >> >> > ________________________________________________________________________ _____ > __ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" > > > -- > Open WebMail Project (http://openwebmail.org) > > ________________________________________________________________________ _______ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are" _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ </pre><font face="monospace"size="-3"><br>The information transmitted is intended only for the person or entity to which it is addressed and <br>may contain confidential and/or privileged material. If the reader of this message is not the intended<br>recipient, you are hereby notified that your access is unauthorized, and any review, dissemination,<br>distribution or copying of this message including any attachments is strictly prohibited. If you are not<br>the intended recipient, please contact the sender and delete the material from any computer.<br><pre> _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug11 www.wwrug.com ARSList: "Where the Answers Are"
