Does this lead anyone to believe or have people noticed other general activities in the applications are slow based on this?
Thanks Peter Lammey ESPN MIT Technical Services & Applications Management 860-766-4761 -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[EMAIL PROTECTED] On Behalf Of Axton Sent: Tuesday, July 10, 2007 11:47 AM To: [email protected] Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... The issue is widespread, esp. with a remote Oracle db. Our servers are switched with fiber/gb. Even if I have 2 vm's runnning on the same physical host (one db and one ars), I see the same issue. The packet logs do not show a problem with throughput and my switches show no congestion. Axton Grams On 7/10/07, Shellman, David <[EMAIL PROTECTED]> wrote: > > > > When I spoke to support about how long it took the system to start up > they said that start up was chatty between the server app and the > database. With a network link that running near capacity (our case), > flaky router/switch, slow link, etc the length of time to respond > between app server and fb server is increased.. All these situations > are often not apparent with out getting network folks involved. > > Dave > -------------------------- > [EMAIL PROTECTED] (Wireless) > > ----- Original Message ----- > From: Action Request System discussion list(ARSList) > <[email protected]> > To: [email protected] <[email protected]> > Sent: Tue Jul 10 10:19:29 2007 > Subject: Re: ARS7 install with oracle 10G RAC ... the saga continues ... > > > The problem does > > not seem to be in the way that the query to the db, but in the way > the > results are processed. > > I don't know about that. > If the problem was the way that arserverd is processing the results > of the query, why would startup be like < 1 minute when the db is on > the same box? > > My bet is that this has something to do with the way that arserverd > is connecting to oracle ... perhaps incorrect use of the api? All > that waiting between queries really makes me think there's a timeout > somewhere ... like waiting for a local listener to respond, i.e. > looking for the database to be on the same machine, and only querying > remotely when the local connection times out. > > Man, this is why I sometimes hate dealing with proprietary software. > If this was open, you could bet someone would have fixed it by now, > and if they hadn't we'd at least have a fighting chance of fixing it > ourselves. > > thanks again, everyone for all your help. > looks like the only answer for now is the Run-DMC response: "it's > like that, and that's the way it is ... huh!" > > -A ;-) > > On Jul 10, 2007, at 8:30 AM, Axton wrote: > > > Good luck with support (and I sincerely mean that). The problem > does > not seem to be in the way that the query to the db, but in the > way the > results are processed. If the arserver could serialize the > memory > contents as part of the shutdown process, then reload them > on startup, > it could probably be designed to be much more > efficient, but there > would be trade offs with that route. A few that come to mind: > > - storage of the serialized contents (possibly gb's) > - time > required to serialize the contents (slower shutdown) > The trade off > is for a faster startup. > > > > Another method that this could be addressed is to expose some > > configuration parameters in how the current process works: > > - chunk size (would be contingent on memory available and > architecture > limits 32 vs 64 bit) > > Another method is to > streamline the current process; where do the > bottlenecks exist in > the current design? Can they be eliminated? > > > > Just some thoughts... > > > > Axton Grams > > > > On 7/10/07, Tony Worthington <[EMAIL PROTECTED]> wrote: > >> Yup. And the second query (each taking about four minutes on our > >> boxes) >> is: > >> > >> SELECT schemaId,fieldId,groupId,permission FROM field_permissions > >> ORDER BY >> 1 ASC,2 ASC,3 ASC >> >> I wonder if we could get a > defect filed, and have this looked at by >> engineering... and get an > answer more than "as designed." > >> > >> -tony > >> > >> -- > >> Tony Worthington > >> [EMAIL PROTECTED] > >> 262-703-5911 > >> > >> > >> > >> Axton <[EMAIL PROTECTED]> > >> Sent by: "Action Request System discussion list(ARSList)" > >> <[email protected]> > >> 07/09/2007 05:55 PM > >> Please respond to > >> [email protected] > >> > >> > >> To > >> [email protected] > >> cc > >> > >> Subject > >> Re: ARS7 install with oracle 10G RAC ... the saga continues ... > >> > >> > >> > >> > >> > >> > >> Remedy fetches information from it's data dictionary (e.g., >> > field_disp_prop, field, etc.), then processes it in chunks of 100, >> > 500, or 1000. So if you have a table field_dispprop that has 120k >> > records and 500 records are processed at a time, 15 seconds per >> > chunk, >> you end up with a start time that totals 3600 seconds (for > that part >> of the startup). > >> > >> If you add a -t parameter to arserverd in the armonitor.conf file, > a >> special startup log file is created. The log file will look >> > something >> like this: > >> > >> Jul 09 2007 18:29:06.6577 <Startup> <TID:3086968512> Set up thread > >> control block with key = 0 >> Jul 09 2007 18:29:06.6582 <Startup> > <TID:3086968512> Initialize >> thread >> local storage block >> Jul > 09 2007 18:29:06.6583 <Startup> <TID:3086968512> Initialize >> > mutiple-byte environment >> Jul 09 2007 18:29:06.6585 <Startup> > <TID:3086968512> InstallDir = >> /u01/arsystem/arsdev >> Jul 09 2007 > 18:29:06.6587 <Startup> <TID:3086968512> Initialize >> Server >> > utility >> Jul 09 2007 18:29:06.6599 <Startup> <TID:3086968512> > Initialize >> License >> Library >> Jul 09 2007 18:29:06.6602 > <Startup> <TID:3086968512> >> LicenseFilename = >> > /etc/arsystem/arsdev/arsystem.lic >> Jul 09 2007 18:29:06.6603 > <Startup> <TID:3086968512> Initialize >> Language setting and locale > >> Jul 09 2007 18:29:06.6610 <Startup> <TID:3086968512> Initialize the > >> Decimal Math libray >> Jul 09 2007 18:29:06.6610 <Startup> > <TID:3086968512> Open shared >> catalog >> Jul 09 2007 18:29:06.6648 > <Startup> <TID:3086968512> Load encryption >> shared library >> Jul > 09 2007 18:29:06.6653 <Startup> <TID:3086968512> Load encryption >> > static functions >> Jul 09 2007 18:29:06.6654 <Startup> > <TID:3086968512> Initialize >> default configuration information >> > Jul 09 2007 18:29:06.6654 <Startup> <TID:3086968512> Create Mutexes > >> Jul 09 2007 18:29:06.6655 <Startup> <TID:3086968512> Initialize > parse >> environment >> Jul 09 2007 18:29:06.6657 <Startup> > <TID:3086968512> Initialize date >> time information >> Jul 09 2007 > 18:29:06.6659 <Startup> <TID:3086968512> Initialize >> notification > strings >> Jul 09 2007 18:29:06.6659 <Startup> <TID:3086968512> > Initialize RPC >> queue type strings >> Jul 09 2007 18:29:06.6660 > <Startup> <TID:3086968512> Initialize >> filter >> strings >> Jul > 09 2007 18:29:06.6662 <Startup> <TID:3086968512> Initialize >> > escalation strings >> Jul 09 2007 18:29:06.6662 <Startup> > <TID:3086968512> Load system >> configuration file >> Jul 09 2007 > 18:29:06.6669 <Startup> <TID:3086968512> >> > arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] >> Jul 09 2007 > 18:29:06.6670 <Startup> <TID:3086968512> Initialize >> pending >> > lists >> Jul 09 2007 18:29:06.6671 <Startup> <TID:3086968512> Create > cache >> read >> write lock >> Jul 09 2007 18:29:06.6671 <Startup> > <TID:3086968512> Create full text >> status read write lock >> Jul > 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Initialize >> fork > proxy >> Jul 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Check > licensing >> Jul 09 2007 18:29:06.6693 <Startup> <TID:3086968512> > Initialize >> user cache >> Jul 09 2007 18:29:06.6707 <Startup> > <TID:3086968512> Open log file >> Jul 09 2007 18:29:06.6745 <Startup> > <TID:3086968512> Initialize >> XML parser >> Jul 09 2007 > 18:29:06.6782 <Startup> <TID:3086968512> Initialize >> thread >> > list >> Jul 09 2007 18:29:06.6784 <Startup> <TID:3086968512> Check > multiple >> servers >> Jul 09 2007 18:29:06.6787 <Startup> > <TID:3086968512> Initialize dead >> thread list >> Jul 09 2007 > 18:29:06.6790 <Startup> <TID:3086968512> Initialize timed >> calls > >> Jul 09 2007 18:29:06.6792 <Startup> <TID:3086968512> >> > CreateRPCQueue(min=1, max=1, rpc=390600) >> Jul 09 2007 18:29:06.6793 > <Startup> <TID:3086968512> Await cache >> ready... > >> Jul 09 2007 18:29:06.8988 <Startup> <TID:0030571424> >> > InitializeServerCache: LoadInitialSchemaInfo Begin >> Jul 09 2007 > 18:29:11.7241 <Startup> <TID:0030571424> Begin >> LoadDisplayInfoList > >> Jul 09 2007 18:29:25.6076 <Startup> <TID:0030571424> >> > LoadDisplayInfoList: 500 rows >> Jul 09 2007 18:29:42.7008 <Startup> > <TID:0030571424> >> LoadDisplayInfoList: 1000 rows >> >> If you > then review the sql that is sent to the db while the server is >> > unavailable, you will see this statement, followed by silence: > >> > >> SELECT schemaId,fieldId,vuiId,propShort,propLong FROM > field_dispprop >> ORDER BY 1 ASC, 2 ASC, 3 ASC >> >> What is > happening here is that arserverd fetches the field display >> > property info, then processes it 500 records at a time. Depending on > >> the size of your server (number of forms, views, fields, etc), > there >> may be a couple of sql statements that are followed by a pause. > >> > >> What is happening internally is anyone's guess; I would guess that > >> they are processing X rows at a time to operate within certain > memory >> constraints, and 'maybe' in a horribly inefficient way. > Some things >> are for certain though; >> (1) a lot of data is > transferred from the db to the arserver. My >> test >> server, > which has 3293 rows in field_dispprop (pretty much a base >> arserver > install) sends 411K of data over the wire. > >> (2) the field_dispprop data is processed in chunks of 500 (at >> > least on >> Linux). > >> > >> Axton Grams > >> > >> > >> On 7/9/07, Andrew Hicox <[EMAIL PROTECTED]> wrote: > >> > ** > >> > Hello everyone: > >> > > >> > I've managed to get through the (incredibly flawed) solaris 10 > >> arsystem >> 7 >> > install. > >> > This is 7.0.01 patch 2. > >> > > >> > I'm using oracle 10G client library with a remote database on an > >> oracle >> 10 >> > Rapid Application Cluster. > >> > > >> > So ... now the server is INCREDIBLY slow to start. > >> > > >> > The server is configured to run on a TCP port (10000). > >> > When I issue the 'arsystem start' command, I get the usual >> > output on the >> > console indicating that the server has started. > >> > > >> > However, the TCP port is not listening, and of course, no >> > clients can >> > connect to the server. > >> > > >> > > >> > If I wait (approx 7 minutes), I will see the following output on > >> the >> > console: > >> > > >> > (ARNOTE 0) > >> > Server indicates that it's up. > >> > > >> > (ARNOTE 0) > >> > ARMonitor child process (pid:29595) started. > >> > ./arplugin > >> > > >> > Action Request System(R) Fork Daemon Version 7.0.01 patch 002 > >> > 200704021644 > >> > Copyright (c) 2000 - 2006 BMC Software, Inc. > >> > All rights reserved. > >> > > >> > Action Request System(R) Plug-In Server Version 7.0.01 patch 002 > >> > 200704021644 > >> > Copyright (c) 2001 - 2006 BMC Software, Inc. > >> > All rights reserved. > >> > Loaded Web Services plugin properly >> > >> > After the > above, the TCP port opens up and clients can connect, >> and it >> > seems >> > to work well enough. > >> > > >> > What I presume is happening here is that arserverd must issue > >> some sort >> of >> > indication that it's completed start up so > that arforkd and >> armonitor >> can do >> > their thing and start > listeners and plugins, and that arserverd >> is for >> some >> > > reason taking on average 7 minutes to do that. > >> > > >> > I can't manage to get ANY sort of logging during the 7 minutes > of >> mystery. > >> > is there some way to figgure out what the heck is going on >> > durring this >> > period of time (is it timing out DNS? is the db > slow to respond? > >> is it > >> > waiting on disk access? what is it doing?). > >> > > >> > We have an identical set up in our dev lab, with the exception > >> that the >> > database server is not a RAC and it's installed on > the same >> machine. The >> > startup is quite snappy on that setup. > >> > > >> > There don't appear to be any issues with network latency between > >> the >> remedy >> > server and the RAC. > >> > > >> > Anyone ever seen this before? > >> > Any suggestions? > >> > > >> > thanks, > >> > > >> > Andrew > >> > >> > _____________________________________________________________________ > >> __________ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> > ARSlist:"Where >> the Answers Are" > >> > >> > >> > >> CONFIDENTIALITY NOTICE: > >> This is a transmission from Kohl's Department Stores, Inc. > >> and may contain information which is confidential and proprietary. > >> If you are not the addressee, any disclosure, copying or >> > distribution or use of the contents of this message is expressly >> > prohibited. > >> If you have received this transmission in error, please destroy it > >> and notify us immediately at 262-703-7000. > >> > >> CAUTION: > >> Internet and e-mail communications are Kohl's property and Kohl's > >> reserves the right to retrieve and read any message created, sent > >> and received. Kohl's reserves the right to monitor messages to or > >> from authorized Kohl's Associates at any time >> without any > further consent. > >> > >> > _____________________________________________________________________ > >> __________ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> > ARSlist:"Where the Answers Are" > >> > > > > > ______________________________________________________________________ > > _________ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > ARSlist:"Where the Answers Are" > > > > ______________________________________________________________________ > _________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > ARSlist:"Where the Answers Are" > > > ________________________________________________________________________ _______ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
