Many thanks to all those who replied to the list and sent me email directly.
After 8 days 12 hours, the issue has been resolved by changing a file system setting on Oracle database server. The setting "mincache" was set to "direct". After we implemented the change at noon today, saving a new incident took only 1-2 seconds, down from over 2 minutes in some cases! The BMC support engineer (thanks to Virginia Burnham) looked at all the loggings I collected and identified that writing BLOBs to disk took too long. The incident table happens to have quite a few BLOB columns. The "mincache" setting was recommended by Oracle/Veritas for Oracle/Solaris platform using Veritas file system. This was a recommendation by Virginia on Tuesday but was not implemented until today after all the evidences showed that BLOG writing to disk took very long time. If you have Oracle/Solaris platform using Veritas file system, you may want to consider setting "mincache" to "direct" with your ITSM 7 implementation. It may save you some sleepless nights. Eric Zhang Sr. IT Consultant Entergy Services, Inc. (501) 377-5815 On Wed, 8 Aug 2007 23:25:34 -0500, ZHANG, ERIC L <[EMAIL PROTECTED]> wrote: >I post this earlier today but don't see it on the list. I am trying >again. I apologize if you already saw this. > >-----Original Message----- >From: Eric Zhang [mailto:[EMAIL PROTECTED] >Sent: Wednesday, August 08, 2007 12:21 PM >To: [EMAIL PROTECTED] >Cc: ZHANG, ERIC L >Subject: URGENT - ITSM 7 performance issue - help needed > >Hello, all. > >We have been struggling with ITSM 7 performance since it went live on >8/1. The performance is getting worse and worse for all the operations: > >search, create, and update. At the begging, it's the creating incident >that took more that 40 seconds to complete. Now it's all the >operations, >primarily on Incident. The incident ticket update can take up to 3 >minutes now. We opened a critical ticket to BMC support but so far we >haven't received a solution that improves the performance. > >Our configurations: > >ARS 7.0.01 no patch on Solaris 9 >Database is Oracle 10gR2 on another Solaris 9 box >Midtier is on Wintel 2003 IIS server (Only for requesters) >ITSM (including Incident and Change) 7.0.02 patch 3 > >I have been sending all kinds of loggings to BMC. The first suggestion >from BMC was to set Oracle-Cursor-Sharing to FORCES. That didn't help. > >The other suggestion was to change a Veritas setting ("mincache=direct") > >on the database server. But our DBA, UNIX. and Storage people all >rejected the idea because 1) that fix is for bulk data import 2) they >had >very experienced with that setting, which produced high IO waits and >worse >overall daily performance. > >Thanks in advance for your help. > >Eric Zhang >Sr. IT Consultant >Entergy Services, Inc. >(501) 377-5815 > >_______________________________________________________________________ ___ _____ >UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are" >======================================================================= = _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
