I finally was able to reproduce this issue. Gavin, thanks very much for your post as this gave me some important clues as to the issue. I believe it is related to the same bug you experienced with version 6.3 patch 19 because the symptoms are almost identical.
My issue was caused by the CCM Calendar View form, which is a link on the home page next to the Change Management Console. When users click on this link and close the window or use the browser back button before the flashboard loads, the issue occurs and 3 threads spike on the arserver process and become hung. I was only able to track this down by noticing that users were accessing this form through api calls during a small time window when I knew that the issue occurred from my information in the windows performance log. I also noticed that some subsequent api calls accessed the Time Segments form which I knew was related to the earlier bug, so that caught my attention. BMC has not provided any fix yet, but I will be hiding this entry point as an easy fix because we do not have any critical business need for the form. On Apr 18, 4:15 am, Gavin Coleman <[EMAIL PROTECTED]> wrote: > I have seen this issue on a Windows 2003 server running ARS 6.3 on Patch > 19. We eventually tracked down the issue as follows: > > 1. Buy / Trial a product called Spotlight. This enables you to tell which > thread is Taking up the CPU on the server. This was what was causing our > CPU to jump. > > 2. At the same time, use a logging by thread for Server side logging. > > 3. Once the CPU spikes and stays high, look at Spotlight and find the > thread that's hanging (or locked). Using this thread number, cross > reference with the Server Side logs. Examine these to try and determine > what was causing the problem > > Our problem was related to the functionality Business-Time-Add. Whenever > this was called on a Windows Server, the CPU jumped by about 10% each time > and locked the thread. It took a long time to persuade Remedy Support that > this was the cause of the problem and we had to do a lot of our own > investigation. Once we discovered this problem, we rolled back to Patch 18 > and this solved the problem. We were in the process of shifting all our > servers from Unix boxes to Windows boxes and this meant the project paused > for nearly 6 weeks before we identified the problem. > > Hope this helps! > > Gavin Coleman > Senior Analyst/Programmer > Computacenter (UK) Ltd > Services & Solutions > Hatfield Avenue > Hatfield, Hertfordshire, AL10 9TW, United Kingdom > T: +44 (0) 1707 631662 > E: [EMAIL PROTECTED] > W:www.computacenter.com > > googerb <[EMAIL PROTECTED]> > Sent by: "Action Request System discussion list(ARSList)" > <[EMAIL PROTECTED]> > 17/04/2008 19:23 > Please respond to arslist > > To: [EMAIL PROTECTED] > cc: > Subject: Re: Arserver.exe high CPU usage > > I was hoping someone had seen a similar issue in the past. Normally > the system will run find for 1-3 days before a spike occurs. By > looking at the windows performance log, I can tell that it is an > abrupt spike that never comes down. Each thread that spikes seems to > take about 10% of the CPU. I have seen up to 7 threads that have > spiked and have taken upwards of 70-80% of the CPU. Eventually the > arserver will crash when this happens. > > Nothing jumps out at me by looking at the arerror log. I have gone as > far as setting up "log per thread" and enabling SQL, API, and Filter > logging, then tracking down the exact time a spike occurs and on which > thread, and cross referencing that to the SQL, API, and Filter logs > for that thread, but still nothing jumps out at me. The API calls are > the same ones that are called quite often but don't result in the same > symptoms. > > I have tried several different combinations of min & max List > threads. Currently the system has 9 min and 9 max list threads (this > configuration was recommended in an AIE performance tuning white paper > for a server with 3 processors). > > Max filters for an operation is 200000 and Max stack of filters is > 10000. Is there any way to identify a filter loop other than by > looking in the logs? > > Also another interesting thing to note is that actual end user > performance does not seem to be affected, however the issue certianly > detracts from overall system stability because the system crashes > every couple of days. Also, when a spike has happened, any admin > action such as saving a filter or form will usually crash the system. > > On Apr 17, 11:43 am, googerb <[EMAIL PROTECTED]> wrote: > > > > > Has anyone experienced issues with sustained, high CPU usage for the > > arserver.exe process? > > > Using: > > ARServer V7.0.1 patch 006 > > SQL server v9 > > Windows Server 2003 Enterprise > > 16 GB of RAM on server > > 3 quad core processors on IBM 3850 xSeries > > > Windows performance monitor with thread logging shows that some > > arserver.exe list threads spike after an unknown amount of time and > > never come down.The only way to resolve the CPU spike is to cycle the > > arserver. Remedy support has been unable to identify a root cause. I > > have had a ticket open with them for 6 months for the issue. Help! > > Thanks! > > _______________________________________________________________________________ > > > UNSUBSCRIBE or access ARSlist Archives atwww.arslist.org > > Platinum Sponsor:www.rmsportal.comARSlist:"Where the Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives atwww.arslist.org > Platinum Sponsor:www.rmsportal.comARSlist: "Where the Answers Are" > > ********************************************************************** > COMPUTACENTER PLC is registered in England and Wales with the registered > number 03110569. Its registered office is at Hatfield Business Park, > Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW > COMPUTACENTER (UK) Limited is registered in England and Wales with the > registered number 01584718. Its registered office is at Hatfield Business > Park, Hatfield Avenue, Hatfield, Hertfordshire AL10 9TW > > The contents of this email are intended for the named addressee only. > It contains information which may be confidential and which may also be > privileged. > Unless you are the named addressee (or authorised to receive mail for the > addressee) you may not copy or use it, or disclose it to anyone else. > > If you receive it in error please notify us immediately and then destroy it. > > Computacenter information is available from:http://www.computacenter.com > ********************************************************************** > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives atwww.arslist.org > Platinum Sponsor:www.rmsportal.comARSlist: "Where the Answers Are"- Hide > quoted text - > > - Show quoted text - _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
