Classification: UNCLASSIFIED
Caveats: NONE
We have been having performance issues with our Oracle installation and
found this KM article with some ideas on how to improve performance and
initial configuration:
*********************************************
The following is information for ITSM 7.0.x installs within an Oracle
environment:
1. Have your DBA set the SGA Size to min 250 and max 850.
2. Also, Knowledge Base entry KM-000010001381 should prove to be
helpful to you:
*********************************************
PROBLEM:
Below are some settings which have been helpful for Oracle customers to
avoid ARERR 91, 92, 93, 94 timeout issues, especially during application
installations
SOLUTION:
1) AR Server version 7.01 or AR Server version 6.03 patch 19
The above version or later contain new method of Oracle client
calls which enhance performance
2) For Oracle 9i or 10g customer:
add CURSOR_SHARING=FORCE to the Oracle init file,
3) For Oracle 9i or 10g customer: add the following parameters to
ar.conf or ar.cfg
Oracle-Cursor-Sharing: FORCE
4) For all Windows (with 2 CPU) customers, Suggest to assign
private RPC to RE (in ar.conf or ar.cfg)
RE-RPC-Socket: 390621
Private-RPC-Socket: 390621 10 10
For UNIX platforms,
RE-RPC-Socket: 390621
Private-RPC-Socket: 390621 6 6
As always, turning of your database and ARServer is a task for your dba,
system administrator and Remedy administrator working in concert. The
settings above have proved helpful in several cases, but they should be
used as a starting point for investigation rather than standard
configuration applicable in all environments.
A defect SW00254374 has been logged for Engineering to consider whether
this information or an official recommendation should be included in the
shipped product docs.
**********
Scott.
-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:[EMAIL PROTECTED] On Behalf Of Lammey, Peter A.
Sent: Tuesday, July 10, 2007 11:51 AM
To: [email protected]
Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ...
Does this lead anyone to believe or have people noticed other general
activities in the applications are slow based on this?
Thanks
Peter Lammey
ESPN MIT Technical Services & Applications Management
860-766-4761
-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:[EMAIL PROTECTED] On Behalf Of Axton
Sent: Tuesday, July 10, 2007 11:47 AM
To: [email protected]
Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ...
The issue is widespread, esp. with a remote Oracle db. Our servers are
switched with fiber/gb. Even if I have 2 vm's runnning on the same
physical host (one db and one ars), I see the same issue. The packet
logs do not show a problem with throughput and my switches show no
congestion.
Axton Grams
On 7/10/07, Shellman, David <[EMAIL PROTECTED]> wrote:
>
>
>
> When I spoke to support about how long it took the system to start up
> they said that start up was chatty between the server app and the
> database. With a network link that running near capacity (our case),
> flaky router/switch, slow link, etc the length of time to respond
> between app server and fb server is increased.. All these situations
> are often not apparent with out getting network folks involved.
>
> Dave
> --------------------------
> [EMAIL PROTECTED] (Wireless)
>
> ----- Original Message -----
> From: Action Request System discussion list(ARSList)
> <[email protected]>
> To: [email protected] <[email protected]>
> Sent: Tue Jul 10 10:19:29 2007
> Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues
...
>
> > The problem does
> > not seem to be in the way that the query to the db, but in the way
> the > results are processed.
>
> I don't know about that.
> If the problem was the way that arserverd is processing the results
> of the query, why would startup be like < 1 minute when the db is on
> the same box?
>
> My bet is that this has something to do with the way that arserverd
> is connecting to oracle ... perhaps incorrect use of the api? All that
> waiting between queries really makes me think there's a timeout
> somewhere ... like waiting for a local listener to respond, i.e.
> looking for the database to be on the same machine, and only querying
> remotely when the local connection times out.
>
> Man, this is why I sometimes hate dealing with proprietary software.
> If this was open, you could bet someone would have fixed it by now,
> and if they hadn't we'd at least have a fighting chance of fixing it
> ourselves.
>
> thanks again, everyone for all your help.
> looks like the only answer for now is the Run-DMC response: "it's
> like that, and that's the way it is ... huh!"
>
> -A ;-)
>
> On Jul 10, 2007, at 8:30 AM, Axton wrote:
>
> > Good luck with support (and I sincerely mean that). The problem
> does > not seem to be in the way that the query to the db, but in the
> way the > results are processed. If the arserver could serialize the
> memory > contents as part of the shutdown process, then reload them
> on startup, > it could probably be designed to be much more
> efficient, but there > would be trade offs with that route. A few
that come to mind:
> > - storage of the serialized contents (possibly gb's) > - time
> required to serialize the contents (slower shutdown) > The trade off
> is for a faster startup.
> >
> > Another method that this could be addressed is to expose some >
> configuration parameters in how the current process works:
> > - chunk size (would be contingent on memory available and
> architecture > limits 32 vs 64 bit) > > Another method is to
> streamline the current process; where do the > bottlenecks exist in
> the current design? Can they be eliminated?
> >
> > Just some thoughts...
> >
> > Axton Grams
> >
> > On 7/10/07, Tony Worthington <[EMAIL PROTECTED]> wrote:
> >> Yup. And the second query (each taking about four minutes on our
> >> boxes) >> is:
> >>
> >> SELECT schemaId,fieldId,groupId,permission FROM field_permissions
> >> ORDER BY >> 1 ASC,2 ASC,3 ASC >> >> I wonder if we could get a
> defect filed, and have this looked at by >> engineering... and get an
> answer more than "as designed."
> >>
> >> -tony
> >>
> >> --
> >> Tony Worthington
> >> [EMAIL PROTECTED]
> >> 262-703-5911
> >>
> >>
> >>
> >> Axton <[EMAIL PROTECTED]>
> >> Sent by: "Action Request System discussion list(ARSList)"
> >> <[email protected]>
> >> 07/09/2007 05:55 PM
> >> Please respond to
> >> [email protected]
> >>
> >>
> >> To
> >> [email protected]
> >> cc
> >>
> >> Subject
> >> Re: ARS7 install with oracle 10G RAC ... the saga continues ...
> >>
> >>
> >>
> >>
> >>
> >>
> >> Remedy fetches information from it's data dictionary (e.g., >>
> field_disp_prop, field, etc.), then processes it in chunks of 100, >>
> 500, or 1000. So if you have a table field_dispprop that has 120k >>
> records and 500 records are processed at a time, 15 seconds per >>
> chunk, >> you end up with a start time that totals 3600 seconds (for
> that part >> of the startup).
> >>
> >> If you add a -t parameter to arserverd in the armonitor.conf file,
> a >> special startup log file is created. The log file will look >>
> something >> like this:
> >>
> >> Jul 09 2007 18:29:06.6577 <Startup> <TID:3086968512> Set up thread
> >> control block with key = 0 >> Jul 09 2007 18:29:06.6582 <Startup>
> <TID:3086968512> Initialize >> thread >> local storage block >> Jul
> 09 2007 18:29:06.6583 <Startup> <TID:3086968512> Initialize >>
> mutiple-byte environment >> Jul 09 2007 18:29:06.6585 <Startup>
> <TID:3086968512> InstallDir = >> /u01/arsystem/arsdev >> Jul 09 2007
> 18:29:06.6587 <Startup> <TID:3086968512> Initialize >> Server >>
> utility >> Jul 09 2007 18:29:06.6599 <Startup> <TID:3086968512>
> Initialize >> License >> Library >> Jul 09 2007 18:29:06.6602
> <Startup> <TID:3086968512> >> LicenseFilename = >>
> /etc/arsystem/arsdev/arsystem.lic >> Jul 09 2007 18:29:06.6603
> <Startup> <TID:3086968512> Initialize >> Language setting and locale
> >> Jul 09 2007 18:29:06.6610 <Startup> <TID:3086968512> Initialize the
> >> Decimal Math libray >> Jul 09 2007 18:29:06.6610 <Startup>
> <TID:3086968512> Open shared >> catalog >> Jul 09 2007 18:29:06.6648
> <Startup> <TID:3086968512> Load encryption >> shared library >> Jul
> 09 2007 18:29:06.6653 <Startup> <TID:3086968512> Load encryption >>
> static functions >> Jul 09 2007 18:29:06.6654 <Startup>
> <TID:3086968512> Initialize >> default configuration information >>
> Jul 09 2007 18:29:06.6654 <Startup> <TID:3086968512> Create Mutexes
> >> Jul 09 2007 18:29:06.6655 <Startup> <TID:3086968512> Initialize
> parse >> environment >> Jul 09 2007 18:29:06.6657 <Startup>
> <TID:3086968512> Initialize date >> time information >> Jul 09 2007
> 18:29:06.6659 <Startup> <TID:3086968512> Initialize >> notification
> strings >> Jul 09 2007 18:29:06.6659 <Startup> <TID:3086968512>
> Initialize RPC >> queue type strings >> Jul 09 2007 18:29:06.6660
> <Startup> <TID:3086968512> Initialize >> filter >> strings >> Jul
> 09 2007 18:29:06.6662 <Startup> <TID:3086968512> Initialize >>
> escalation strings >> Jul 09 2007 18:29:06.6662 <Startup>
> <TID:3086968512> Load system >> configuration file >> Jul 09 2007
> 18:29:06.6669 <Startup> <TID:3086968512> >>
> arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] >> Jul 09 2007
> 18:29:06.6670 <Startup> <TID:3086968512> Initialize >> pending >>
> lists >> Jul 09 2007 18:29:06.6671 <Startup> <TID:3086968512> Create
> cache >> read >> write lock >> Jul 09 2007 18:29:06.6671 <Startup>
> <TID:3086968512> Create full text >> status read write lock >> Jul
> 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Initialize >> fork
> proxy >> Jul 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Check
> licensing >> Jul 09 2007 18:29:06.6693 <Startup> <TID:3086968512>
> Initialize >> user cache >> Jul 09 2007 18:29:06.6707 <Startup>
> <TID:3086968512> Open log file >> Jul 09 2007 18:29:06.6745 <Startup>
> <TID:3086968512> Initialize >> XML parser >> Jul 09 2007
> 18:29:06.6782 <Startup> <TID:3086968512> Initialize >> thread >>
> list >> Jul 09 2007 18:29:06.6784 <Startup> <TID:3086968512> Check
> multiple >> servers >> Jul 09 2007 18:29:06.6787 <Startup>
> <TID:3086968512> Initialize dead >> thread list >> Jul 09 2007
> 18:29:06.6790 <Startup> <TID:3086968512> Initialize timed >> calls
> >> Jul 09 2007 18:29:06.6792 <Startup> <TID:3086968512> >>
> CreateRPCQueue(min=1, max=1, rpc=390600) >> Jul 09 2007 18:29:06.6793
> <Startup> <TID:3086968512> Await cache >> ready...
> >> Jul 09 2007 18:29:06.8988 <Startup> <TID:0030571424> >>
> InitializeServerCache: LoadInitialSchemaInfo Begin >> Jul 09 2007
> 18:29:11.7241 <Startup> <TID:0030571424> Begin >> LoadDisplayInfoList
> >> Jul 09 2007 18:29:25.6076 <Startup> <TID:0030571424> >>
> LoadDisplayInfoList: 500 rows >> Jul 09 2007 18:29:42.7008 <Startup>
> <TID:0030571424> >> LoadDisplayInfoList: 1000 rows >> >> If you
> then review the sql that is sent to the db while the server is >>
> unavailable, you will see this statement, followed by silence:
> >>
> >> SELECT schemaId,fieldId,vuiId,propShort,propLong FROM
> field_dispprop >> ORDER BY 1 ASC, 2 ASC, 3 ASC >> >> What is
> happening here is that arserverd fetches the field display >>
> property info, then processes it 500 records at a time. Depending on
> >> the size of your server (number of forms, views, fields, etc),
> there >> may be a couple of sql statements that are followed by a
pause.
> >>
> >> What is happening internally is anyone's guess; I would guess that
> >> they are processing X rows at a time to operate within certain
> memory >> constraints, and 'maybe' in a horribly inefficient way.
> Some things >> are for certain though; >> (1) a lot of data is
> transferred from the db to the arserver. My >> test >> server,
> which has 3293 rows in field_dispprop (pretty much a base >> arserver
> install) sends 411K of data over the wire.
> >> (2) the field_dispprop data is processed in chunks of 500 (at >>
> least on >> Linux).
> >>
> >> Axton Grams
> >>
> >>
> >> On 7/9/07, Andrew Hicox <[EMAIL PROTECTED]> wrote:
> >> > **
> >> > Hello everyone:
> >> >
> >> > I've managed to get through the (incredibly flawed) solaris 10
> >> arsystem >> 7 >> > install.
> >> > This is 7.0.01 patch 2.
> >> >
> >> > I'm using oracle 10G client library with a remote database on an
> >> oracle >> 10 >> > Rapid Application Cluster.
> >> >
> >> > So ... now the server is INCREDIBLY slow to start.
> >> >
> >> > The server is configured to run on a TCP port (10000).
> >> > When I issue the 'arsystem start' command, I get the usual >>
> output on the >> > console indicating that the server has started.
> >> >
> >> > However, the TCP port is not listening, and of course, no >>
> clients can >> > connect to the server.
> >> >
> >> >
> >> > If I wait (approx 7 minutes), I will see the following output on
> >> the >> > console:
> >> >
> >> > (ARNOTE 0)
> >> > Server indicates that it's up.
> >> >
> >> > (ARNOTE 0)
> >> > ARMonitor child process (pid:29595) started.
> >> > ./arplugin
> >> >
> >> > Action Request System(R) Fork Daemon Version 7.0.01 patch 002
> >> > 200704021644
> >> > Copyright (c) 2000 - 2006 BMC Software, Inc.
> >> > All rights reserved.
> >> >
> >> > Action Request System(R) Plug-In Server Version 7.0.01 patch
002
> >> > 200704021644
> >> > Copyright (c) 2001 - 2006 BMC Software, Inc.
> >> > All rights reserved.
> >> > Loaded Web Services plugin properly >> > >> > After the
> above, the TCP port opens up and clients can connect, >> and it >>
> seems >> > to work well enough.
> >> >
> >> > What I presume is happening here is that arserverd must issue
> >> some sort >> of >> > indication that it's completed start up so
> that arforkd and >> armonitor >> can do >> > their thing and start
> listeners and plugins, and that arserverd >> is for >> some >> >
> reason taking on average 7 minutes to do that.
> >> >
> >> > I can't manage to get ANY sort of logging during the 7 minutes
> of >> mystery.
> >> > is there some way to figgure out what the heck is going on >>
> durring this >> > period of time (is it timing out DNS? is the db
> slow to respond?
> >> is it
> >> > waiting on disk access? what is it doing?).
> >> >
> >> > We have an identical set up in our dev lab, with the exception
> >> that the >> > database server is not a RAC and it's installed on
> the same >> machine. The >> > startup is quite snappy on that setup.
> >> >
> >> > There don't appear to be any issues with network latency between
> >> the >> remedy >> > server and the RAC.
> >> >
> >> > Anyone ever seen this before?
> >> > Any suggestions?
> >> >
> >> > thanks,
> >> >
> >> > Andrew
> >>
> >>
> _____________________________________________________________________
> >> __________
> >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >>
> ARSlist:"Where >> the Answers Are"
> >>
> >>
> >>
> >> CONFIDENTIALITY NOTICE:
> >> This is a transmission from Kohl's Department Stores, Inc.
> >> and may contain information which is confidential and proprietary.
> >> If you are not the addressee, any disclosure, copying or >>
> distribution or use of the contents of this message is expressly >>
> prohibited.
> >> If you have received this transmission in error, please destroy it
> >> and notify us immediately at 262-703-7000.
> >>
> >> CAUTION:
> >> Internet and e-mail communications are Kohl's property and Kohl's
> >> reserves the right to retrieve and read any message created, sent
> >> and received. Kohl's reserves the right to monitor messages to or
> >> from authorized Kohl's Associates at any time >> without any
> further consent.
> >>
> >>
> _____________________________________________________________________
> >> __________
> >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >>
> ARSlist:"Where the Answers Are"
> >>
> >
> >
> ______________________________________________________________________
> > _________
> > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >
> ARSlist:"Where the Answers Are"
> >
>
> ______________________________________________________________________
> _________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
> ARSlist:"Where the Answers Are"
>
>
>
________________________________________________________________________
_______
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where
the Answers Are"
________________________________________________________________________
_______
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where
the Answers Are"
Classification: UNCLASSIFIED
Caveats: NONE
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the
Answers Are"
