API, SQL, and filter. Make a copy of them before you restart arserver so they are not overwritten or otherwise lost. See what the last entries in the logs are at the time it was hung. This will most likely point you to the cause. These log entries are just symptomatic of the underlying issue: - AssignEng : Timeout during database update - SLMCS : Timeout during database update - BRIE : Timeout during database query - CMDB Dispatcher : Timeout during database query With the underlying issue being that your app server and/or db server are unresponsive. When it says unresponsive, it usually means a rather long time, I think 5 minutes, if my memory serves me correctly.
If you inspect the SQL logs, you should be able to get a list of the last sql statements issued to the database server. Look for the one(s) that have not returned. If all the threads are blocked, you will get this error. Take a look at the server statistics and see if you have any queue blocked items. Check the Cache-Mode of your server and make sure it is in production mode. Escalations can cause the server to hang in this way if the server is in Dev cache mode. Not sure what OS you are using, but gathering a stack trace for all the threads for arserver can be helpful in seeing what arserver is doing at the time it is hung. If possible, generate a core dump from the arserver process before restarting. This will let you dig into the state of the process at a later time, after restoring services. Axton Grams On Tue, Feb 14, 2012 at 2:26 PM, Avy Mann <[email protected]> wrote: > ** > Ron, > > I wanted to suggest looking at any actions that trigger at 6 am. > Escalations come to mind because I noticed a line in your log: > > Fri Feb 10 06:30:05 2012 BRIE : Timeout during database query -- consider > using more specific search criteria to narrow the results, and retry the > operation (OUR_AR_SERVER) ARERR - 94 > > I see this repeated. I am thinking there maybe be something in the > background running a bad query. Just a thought. > > > ________________________________ > From: "Peters, Ron" <[email protected]> > To: [email protected] > Sent: Tuesday, February 14, 2012 3:12:25 PM > > Subject: Re: AR Server or DB issue > > ** > Hmm, that KB article is interesting. It sure does sound plausible. If the > server is stuck in a loop, resetting its brain seems like the right > solution. Since the server is unresponsive during our issues, what should I > look for on the DB to see what is causing the issue? > > I’m working with the DBA on this and since it happened last week mon/wed/fri > and again yesterday, I can reasonably assume it will happen again tomorrow > morning. I’d like to set up some type of process to catch this in the act > and I’d like to gather all the appropriate data. Thougths? > > Thanks. > > From: Action Request System discussion list(ARSList) > [mailto:[email protected]] On Behalf Of Jonas Stumph Stevnsvig > Sent: Monday, February 13, 2012 2:47 PM > To: [email protected] > Subject: Re: AR Server or DB issue > > ** > Hi Ron > > I noticed the arAPPnote - indicating an approval notice. I've recently > hotfxed our servers which were presenting us with ARERR 93 (server busy) or > the same error you are getting (ARERR 92). Our servers would hang upon > handling some approval actions. The symtom turned out to be unbelievable > numbers of notifications being triggered - by a single Approval on a change. > We got 130000 entries in NTE:SYS-NT Proces Control for a single change. > hotfix was SW00369891 Knowledgebase article is KA 354923 - just a hunch - > but check your NTE:SYS-NT Proces Control form (or select from the Database > if your server is unresponsive as ours was). > > br > > Jonas > > Den 10-02-2012 16:34, Peters, Ron skrev: > ** > Hello all, > > Having some AR server issues. Something has been happening in the mornings > about 6am in that our service desk can’t access Remedy. This started in > Decemeber but only once. Then twice in January and now a couple times this > week. The first few times it resolved itself after about 30 minutes but this > week we had to restart the AR service on the server to get it back. The > following is the arerror log from this morning. I’ve contacted our DBA’s to > see if they notice anything but since an AR restart resolves it, I’m not > convinced it’s a DB issue. Any thoughts? > > Thanks. > > Fri Feb 10 06:25:30 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:25:30 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:25:31 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:25:31 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:27:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:27:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:27:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:27:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:29:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:29:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:29:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:29:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:30:05 2012 BRIE : Timeout during database query -- consider > using more specific search criteria to narrow the results, and retry the > operation (OUR_AR_SERVER) ARERR - 94 > Fri Feb 10 06:31:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:31:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:31:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:31:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:33:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:33:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:33:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:33:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:35:02 2012 CMDB Dispatcher : Timeout during database query -- > consider using more specific search criteria to narrow the results, and > retry the operation (OUR_AR_SERVER) ARERR - 94 > Fri Feb 10 06:35:05 2012 BRIE : Timeout during database query -- consider > using more specific search criteria to narrow the results, and retry the > operation (OUR_AR_SERVER) ARERR - 94 > Fri Feb 10 06:35:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:35:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:35:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:35:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:37:32 2012 AssignEng : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:37:32 2012 AssignEng : AR System Application server > terminated -- fatal error encountered (ARAPPNOTE 4501) > Fri Feb 10 06:37:32 2012 SLMCS : Timeout during database update -- the > operation has been accepted by the server and will usually complete > successfully (OUR_AR_SERVER) ARERR - 92 > Fri Feb 10 06:37:32 2012 SLMCS : AR System Application server terminated -- > fatal error encountered (ARAPPNOTE 4501) > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
