After intense troubleshooting on this server with our Remedy, NT Server, DB, Network, and UNIX teams I believe that we have solved this issue. 1.) We found that Oracle 11g was running Stat Reports at 10pm on all databases so there were 18 jobs running on the db that was putting the load up to 15 (20 is max capacity?). 2.) Automated server backups were kicking off around the same time (give or take an hour) and was running for almost 2 hours. This was also transferring around 35 to 40 gb of data to our Tivoli server. 3.) Initial troubleshooting on our part was to disable Developer Cache Mode.
Resolution steps 1.) Reschedule DB jobs. They were running in this manner because it was default OOB configuration, changing that config was not a problem for the dba's. 2.) Set our server to start backups in an earlier time slot. 3.) Re-enabled Dev mode. 4.) Disabled escalations There were no reported performance issues last night. Thanks to everyone for the suggestions. I actually didn't even look at the dev mode after disabling it as I was focusing more on the server itself. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of patchsk Sent: Saturday, February 06, 2010 2:52 AM To: [email protected] Subject: Re: RPC Timeout Errors Turning on devcache mode will increase the performance on a dev env. But with just one/two developers doing code changes I don't think it is the real cause. RPC errors usually due to server crashes are due to server busy. Some places to look for: 1.Malloc error seems to be some memory related. See if any other processes are running at the same time and taking more ram leaving less to remedy. 2.There seems to be some bottle neck some where taking more time and process might be timing out. Check the SQL logs to see if the db is taking more time for response. Also looking at api logs also will tell you at which operation it is restarting. 3. BMC has released a patch 7.5p3A for devstudio to fix an issue where dev studio is crashing the arserver. 4. If possible stop all other processes (DSO,Email,Archive,Escalations etc..) and try the same thing. ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
