Hey Norm...have your DBA (or you if you would like) turn on profiler on your SQL Server right before you issue your search....and wait for the search to finish (instead of restarting in the middle)...I recently had a problem where periodically my server would just hang....tracked it down to our reporting team doing a select * on a table with about 80 columns and 350K rows...the search was taking over 4 min's to complete....all searches/inserts/updates/anything on that table were queue'd up until that one query was done...took me weeks to track it down to this....we ended up optimizing their query and the problem went away. This unfortunately isn't likely to do anything to solve your problem...but it will show you what's happening at a DB level.
-----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Kaiser, Norm E CIV USAF AFMC 96 CS/SCCE Sent: Wednesday, December 03, 2008 1:18 PM To: arslist@ARSLIST.ORG Subject: UPDATE: Intermittent, Spotty ARS Performance (UNCLASSIFIED) *** UPDATE *** Well, it appears I don't have to toilet paper my own house after all. As I reported below, I attempted to fix my problem with spotty, intermittent performance by increasing my fast and list threads from 5 to 30. That did nothing. I still have the same problem. The issue is most definitely related to diary searches, but I would *expect* that a diary search would not so drastically impact ALL users as it does. I can understand the person who invoked the diary search having a problem, but considering there are so many other threads available and CPU utilization remains under 10%, I would think other users would not be impacted so dramatically. But it never fails...I can reproduce the problem 100% of the time. I kick off a diary search, and everyone's client stops responding. It's almost like (I know this is dreaded and sometimes over-reported), but it's almost like a memory leak in the server app. Restarting the service, naturally, rectifies the situation and Remedy just hums along until someone else does a diary search. Now, I understand I can block diary searches. But my issue is wondering why diary searches impact EVERYONE. Ideas? Norm -----Original Message----- From: Kaiser, Norm E CIV USAF AFMC 96 CS/SCCE Sent: Tuesday, November 25, 2008 5:39 PM To: 'arslist@ARSLIST.ORG' Subject: RE: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Well, a little bird (ahem...cough...Doug...cough) suggested I double up my list and fast threads, which I've done, and that seems -- at least on the surface -- to have corrected the problem. I did have multiple threads, but I guess just not enough. If that was the problem, I'm going to toilet paper my own house. Norm -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Mayfield, Andy L. Sent: Tuesday, November 25, 2008 5:02 PM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Never mind...... I guess I should read the entire thread before responding. Andy L. Mayfield Sr. System Operation Specialist Alabama Power Company Office: 205-226-1805 Cell: 205-288-9140 SoLinc: 10*19140 -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Mayfield, Andy L. Sent: Tuesday, November 25, 2008 4:17 PM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) We had a similar problem recently. We found that an Active Link was causing our problem. It was a newly created Active Link that was somehow corrupted and caused the server to hang. It might be worth a look. Check to see if any objects have been created or modified recently. Good Luck. Andy L. Mayfield Sr. System Operation Specialist Alabama Power Company Office: 205-226-1805 Cell: 205-288-9140 SoLinc: 10*19140 -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Michaud, Christopher W Mr CTR USA MEDCOM USAMITC Sent: Tuesday, November 25, 2008 9:27 AM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Classification: UNCLASSIFIED Caveats: NONE Norm, You may want to investigate whether you can use BMC or SQL Full Text search options to improve the performance. Alternatively, I've found it helps to interview the culprits to understand how they are utilizing the system to do their job. Often you can add an indexed field that allows them to categorize/track what they are looking for on a repeat basis. Christopher Michaud -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Kaiser, Norm E CIV USAF AFMC 96 CS/SCCE Sent: Tuesday, November 25, 2008 8:25 AM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Good suggestion...I'm pretty familiar with the new worklog model in version 7 and its advantages and disadvantages. Unfortunately, that entails a very large coding effort, which I'm not able to do on this system. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Benedetto Cantatore Sent: Tuesday, November 25, 2008 8:12 AM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) ** Norm, Perhaps you need to steal an idea from version 7 and make the worklogs a parent-child relationship with the main form. This would accomodate the individuals that need to get to specific information in the worklog and ease up the burden on your database. If you can install version 7 on a server, you'll see how it works and adopt it. Ben Cantatore Remedy Manager (914) 457-6209 Emerging Health IT 3 Odell Plaza Yonkers, New York 10701 >>> [EMAIL PROTECTED] 11/25/08 8:56 AM >>> Yeah, I suspected the same thing going in, but free disk space is abundant. Only about 20% of the disk is used. I have concluded that the issue is the diary searches. I suspected that this was a problem about a month ago, so I created a form and a filter that would capture a record every time a user did a diary search. Sure enough, I discovered users were doing diary searches dozens of times per day. There are now over 500,000 tickets in this system, and each ticket contains diary entries of up to 30 pages (or more) in length. Users were repeatedly searching for things like, "The ticket was placed on hold because the customer is unavailable." To prove the theory, I had the administrator at the site repeatedly log on to her User client. That is, TOOL...LOGIN...TOOLS...LOGIN...TOOLS...LOGIN...etc. The User client would faithfully log her on to Remedy in under a second. I told her, "Keep doing it!" while I went to my client and issued a diary search. Bam! She could no longer log in. She got the dreaded, "Setting server port..." message that never went away. So I have locked down the diary field to prevent these searches, but I'm already hearing all sorts of dissent: "That puts us out of business! We HAVE to be able to search the worklog!" So now I'm considering other options. I suppose the only thing I can do is set up some type of archival system, but that comes with two problems: 1) Users will hate it and 2) It doesn't really solve the problem. Putting a voluminous amount of free text on another form and telling users, "Go search there," still puts a huge burden on the database to sift through all that garbage. Norm -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Joe DeSouza Sent: Monday, November 24, 2008 8:09 PM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) ** Another thing could be your disk space getting full on the Remedy server. We had that issue recently when one of the operation some user would do would eventually timeout and would create a temp file on the servers Windows Temp directory that would grow and keep growing even if the user quit the user tool from the client. The disk would eventually be full and the AR Server would get extremely slow and eventually impossible to login. Bouoncing the Remedy Service would kill that temp file and release all the used space.. Joe ________________________________ From: "Kaiser, Norm E CIV USAF AFMC 96 CS/SCCE" <[EMAIL PROTECTED]> To: arslist@ARSLIST.ORG Sent: Monday, November 24, 2008 12:58:53 PM Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Yes, that's my suspicion. I have a big suspicion that people are searching the worklog diary field. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Michaud, Christopher W Mr CTRUSA MEDCOM USAMITC Sent: Monday, November 24, 2008 11:20 AM To: arslist@ARSLIST.ORG Subject: Re: Intermittent, Spotty ARS Performance (UNCLASSIFIED) Classification: UNCLASSIFIED Caveats: NONE Norm, You may want to look closer at the SQL side. Look for locks. Perhaps someone querying a diary or un-indexed field. Also, are you using SQL replication? In particular, are snapshots turned on? Christopher Michaud -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Kaiser, Norm E CIV USAF AFMC96 CS/SCCE Sent: Monday, November 24, 2008 11:03 AM To: arslist@ARSLIST.ORG Subject: Intermittent, Spotty ARS Performance ** Hi everyone: This problem has me perplexed. At a site I support, the Remedy server inexplicably stops responding to requests. It's very intermittent. It runs fine for awhile, then seemingly without warning, it just hangs. Users attempting to log on get stuck at the "Setting server port" dialog, which eventually times out. Other users who are already logged who try to pull up a ticket get stuck at a blank screen that never comes back. To resolve the issue, they have to bounce the Remedy server service. The system works for awhile...until it hangs up again. Any ideas what might be causing this? - I have monitored CPU utilization when this occurs, and the CPU hums along at about 3% - 5% utilization - Network utilization is flat-lined whenever this occurs (i.e., no spike) - Memory utilization appears normal - CNET bandwidth tests resolve to better than dedicated T1 performance (for what that's worth) Any thoughts are greatly appreciated. The interesting thing is, we have the same exact Remedy apps running on the same exact type of server in the same exact environment in four other locations, and those four other locations never experience any problems. Norm Remedy ARS 6.3 Microsoft SQL 2000 SP4 Microsoft Windows 2000 SP2 100% Custom Apps - No ITSM __Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" html___ ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" __Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" html___ ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" Classification: UNCLASSIFIED Caveats: NONE ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" ____________________________________________________________________________ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor: www.rmsportal.com ARSlist: "Where the Answers Are"
