Definitely opaque. All I have to go on are packet dumps, sqlnet logs, low level system calls, and an arserver startup log generated by using an undocumented flag to arserverd.
Axton Grams On 7/10/07, Andrew Hicox <[EMAIL PROTECTED]> wrote:
> The problem does > not seem to be in the way that the query to the db, but in the way the > results are processed. I don't know about that. If the problem was the way that arserverd is processing the results of the query, why would startup be like < 1 minute when the db is on the same box? My bet is that this has something to do with the way that arserverd is connecting to oracle ... perhaps incorrect use of the api? All that waiting between queries really makes me think there's a timeout somewhere ... like waiting for a local listener to respond, i.e. looking for the database to be on the same machine, and only querying remotely when the local connection times out. Man, this is why I sometimes hate dealing with proprietary software. If this was open, you could bet someone would have fixed it by now, and if they hadn't we'd at least have a fighting chance of fixing it ourselves. thanks again, everyone for all your help. looks like the only answer for now is the Run-DMC response: "it's like that, and that's the way it is ... huh!" -A ;-) On Jul 10, 2007, at 8:30 AM, Axton wrote: > Good luck with support (and I sincerely mean that). The problem does > not seem to be in the way that the query to the db, but in the way the > results are processed. If the arserver could serialize the memory > contents as part of the shutdown process, then reload them on startup, > it could probably be designed to be much more efficient, but there > would be trade offs with that route. A few that come to mind: > - storage of the serialized contents (possibly gb's) > - time required to serialize the contents (slower shutdown) > The trade off is for a faster startup. > > Another method that this could be addressed is to expose some > configuration parameters in how the current process works: > - chunk size (would be contingent on memory available and architecture > limits 32 vs 64 bit) > > Another method is to streamline the current process; where do the > bottlenecks exist in the current design? Can they be eliminated? > > Just some thoughts... > > Axton Grams > > On 7/10/07, Tony Worthington <[EMAIL PROTECTED]> wrote: >> Yup. And the second query (each taking about four minutes on our >> boxes) >> is: >> >> SELECT schemaId,fieldId,groupId,permission FROM field_permissions >> ORDER BY >> 1 ASC,2 ASC,3 ASC >> >> I wonder if we could get a defect filed, and have this looked at by >> engineering... and get an answer more than "as designed." >> >> -tony >> >> -- >> Tony Worthington >> [EMAIL PROTECTED] >> 262-703-5911 >> >> >> >> Axton <[EMAIL PROTECTED]> >> Sent by: "Action Request System discussion list(ARSList)" >> <[email protected]> >> 07/09/2007 05:55 PM >> Please respond to >> [email protected] >> >> >> To >> [email protected] >> cc >> >> Subject >> Re: ARS7 install with oracle 10G RAC ... the saga continues ... >> >> >> >> >> >> >> Remedy fetches information from it's data dictionary (e.g., >> field_disp_prop, field, etc.), then processes it in chunks of 100, >> 500, or 1000. So if you have a table field_dispprop that has 120k >> records and 500 records are processed at a time, 15 seconds per >> chunk, >> you end up with a start time that totals 3600 seconds (for that part >> of the startup). >> >> If you add a -t parameter to arserverd in the armonitor.conf file, a >> special startup log file is created. The log file will look >> something >> like this: >> >> Jul 09 2007 18:29:06.6577 <Startup> <TID:3086968512> Set up thread >> control block with key = 0 >> Jul 09 2007 18:29:06.6582 <Startup> <TID:3086968512> Initialize >> thread >> local storage block >> Jul 09 2007 18:29:06.6583 <Startup> <TID:3086968512> Initialize >> mutiple-byte environment >> Jul 09 2007 18:29:06.6585 <Startup> <TID:3086968512> InstallDir = >> /u01/arsystem/arsdev >> Jul 09 2007 18:29:06.6587 <Startup> <TID:3086968512> Initialize >> Server >> utility >> Jul 09 2007 18:29:06.6599 <Startup> <TID:3086968512> Initialize >> License >> Library >> Jul 09 2007 18:29:06.6602 <Startup> <TID:3086968512> >> LicenseFilename = >> /etc/arsystem/arsdev/arsystem.lic >> Jul 09 2007 18:29:06.6603 <Startup> <TID:3086968512> Initialize >> Language setting and locale >> Jul 09 2007 18:29:06.6610 <Startup> <TID:3086968512> Initialize the >> Decimal Math libray >> Jul 09 2007 18:29:06.6610 <Startup> <TID:3086968512> Open shared >> catalog >> Jul 09 2007 18:29:06.6648 <Startup> <TID:3086968512> Load encryption >> shared library >> Jul 09 2007 18:29:06.6653 <Startup> <TID:3086968512> Load encryption >> static functions >> Jul 09 2007 18:29:06.6654 <Startup> <TID:3086968512> Initialize >> default configuration information >> Jul 09 2007 18:29:06.6654 <Startup> <TID:3086968512> Create Mutexes >> Jul 09 2007 18:29:06.6655 <Startup> <TID:3086968512> Initialize parse >> environment >> Jul 09 2007 18:29:06.6657 <Startup> <TID:3086968512> Initialize date >> time information >> Jul 09 2007 18:29:06.6659 <Startup> <TID:3086968512> Initialize >> notification strings >> Jul 09 2007 18:29:06.6659 <Startup> <TID:3086968512> Initialize RPC >> queue type strings >> Jul 09 2007 18:29:06.6660 <Startup> <TID:3086968512> Initialize >> filter >> strings >> Jul 09 2007 18:29:06.6662 <Startup> <TID:3086968512> Initialize >> escalation strings >> Jul 09 2007 18:29:06.6662 <Startup> <TID:3086968512> Load system >> configuration file >> Jul 09 2007 18:29:06.6669 <Startup> <TID:3086968512> >> arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0] >> Jul 09 2007 18:29:06.6670 <Startup> <TID:3086968512> Initialize >> pending >> lists >> Jul 09 2007 18:29:06.6671 <Startup> <TID:3086968512> Create cache >> read >> write lock >> Jul 09 2007 18:29:06.6671 <Startup> <TID:3086968512> Create full text >> status read write lock >> Jul 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Initialize >> fork proxy >> Jul 09 2007 18:29:06.6672 <Startup> <TID:3086968512> Check licensing >> Jul 09 2007 18:29:06.6693 <Startup> <TID:3086968512> Initialize >> user cache >> Jul 09 2007 18:29:06.6707 <Startup> <TID:3086968512> Open log file >> Jul 09 2007 18:29:06.6745 <Startup> <TID:3086968512> Initialize >> XML parser >> Jul 09 2007 18:29:06.6782 <Startup> <TID:3086968512> Initialize >> thread >> list >> Jul 09 2007 18:29:06.6784 <Startup> <TID:3086968512> Check multiple >> servers >> Jul 09 2007 18:29:06.6787 <Startup> <TID:3086968512> Initialize dead >> thread list >> Jul 09 2007 18:29:06.6790 <Startup> <TID:3086968512> Initialize timed >> calls >> Jul 09 2007 18:29:06.6792 <Startup> <TID:3086968512> >> CreateRPCQueue(min=1, max=1, rpc=390600) >> Jul 09 2007 18:29:06.6793 <Startup> <TID:3086968512> Await cache >> ready... >> Jul 09 2007 18:29:06.8988 <Startup> <TID:0030571424> >> InitializeServerCache: LoadInitialSchemaInfo Begin >> Jul 09 2007 18:29:11.7241 <Startup> <TID:0030571424> Begin >> LoadDisplayInfoList >> Jul 09 2007 18:29:25.6076 <Startup> <TID:0030571424> >> LoadDisplayInfoList: 500 rows >> Jul 09 2007 18:29:42.7008 <Startup> <TID:0030571424> >> LoadDisplayInfoList: 1000 rows >> >> If you then review the sql that is sent to the db while the server is >> unavailable, you will see this statement, followed by silence: >> >> SELECT schemaId,fieldId,vuiId,propShort,propLong FROM field_dispprop >> ORDER BY 1 ASC, 2 ASC, 3 ASC >> >> What is happening here is that arserverd fetches the field display >> property info, then processes it 500 records at a time. Depending on >> the size of your server (number of forms, views, fields, etc), there >> may be a couple of sql statements that are followed by a pause. >> >> What is happening internally is anyone's guess; I would guess that >> they are processing X rows at a time to operate within certain memory >> constraints, and 'maybe' in a horribly inefficient way. Some things >> are for certain though; >> (1) a lot of data is transferred from the db to the arserver. My >> test >> server, which has 3293 rows in field_dispprop (pretty much a base >> arserver install) sends 411K of data over the wire. >> (2) the field_dispprop data is processed in chunks of 500 (at >> least on >> Linux). >> >> Axton Grams >> >> >> On 7/9/07, Andrew Hicox <[EMAIL PROTECTED]> wrote: >> > ** >> > Hello everyone: >> > >> > I've managed to get through the (incredibly flawed) solaris 10 >> arsystem >> 7 >> > install. >> > This is 7.0.01 patch 2. >> > >> > I'm using oracle 10G client library with a remote database on an >> oracle >> 10 >> > Rapid Application Cluster. >> > >> > So ... now the server is INCREDIBLY slow to start. >> > >> > The server is configured to run on a TCP port (10000). >> > When I issue the 'arsystem start' command, I get the usual >> output on the >> > console indicating that the server has started. >> > >> > However, the TCP port is not listening, and of course, no >> clients can >> > connect to the server. >> > >> > >> > If I wait (approx 7 minutes), I will see the following output on >> the >> > console: >> > >> > (ARNOTE 0) >> > Server indicates that it's up. >> > >> > (ARNOTE 0) >> > ARMonitor child process (pid:29595) started. >> > ./arplugin >> > >> > Action Request System(R) Fork Daemon Version 7.0.01 patch 002 >> > 200704021644 >> > Copyright (c) 2000 - 2006 BMC Software, Inc. >> > All rights reserved. >> > >> > Action Request System(R) Plug-In Server Version 7.0.01 patch 002 >> > 200704021644 >> > Copyright (c) 2001 - 2006 BMC Software, Inc. >> > All rights reserved. >> > Loaded Web Services plugin properly >> > >> > After the above, the TCP port opens up and clients can connect, >> and it >> seems >> > to work well enough. >> > >> > What I presume is happening here is that arserverd must issue >> some sort >> of >> > indication that it's completed start up so that arforkd and >> armonitor >> can do >> > their thing and start listeners and plugins, and that arserverd >> is for >> some >> > reason taking on average 7 minutes to do that. >> > >> > I can't manage to get ANY sort of logging during the 7 minutes of >> mystery. >> > is there some way to figgure out what the heck is going on >> durring this >> > period of time (is it timing out DNS? is the db slow to respond? >> is it >> > waiting on disk access? what is it doing?). >> > >> > We have an identical set up in our dev lab, with the exception >> that the >> > database server is not a RAC and it's installed on the same >> machine. The >> > startup is quite snappy on that setup. >> > >> > There don't appear to be any issues with network latency between >> the >> remedy >> > server and the RAC. >> > >> > Anyone ever seen this before? >> > Any suggestions? >> > >> > thanks, >> > >> > Andrew >> >> _____________________________________________________________________ >> __________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> ARSlist:"Where >> the Answers Are" >> >> >> >> CONFIDENTIALITY NOTICE: >> This is a transmission from Kohl's Department Stores, Inc. >> and may contain information which is confidential and proprietary. >> If you are not the addressee, any disclosure, copying or >> distribution or use of the contents of this message is expressly >> prohibited. >> If you have received this transmission in error, please destroy it >> and notify us immediately at 262-703-7000. >> >> CAUTION: >> Internet and e-mail communications are Kohl's property and Kohl's >> reserves the right to retrieve and read any message created, sent >> and received. Kohl's reserves the right to monitor messages to or >> from authorized Kohl's Associates at any time >> without any further consent. >> >> _____________________________________________________________________ >> __________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> ARSlist:"Where the Answers Are" >> > > ______________________________________________________________________ > _________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > ARSlist:"Where the Answers Are" > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
_______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org ARSlist:"Where the Answers Are"
