Hi Brad, It would be cool to meet you in person. I didn't notice in your original email you already had turned on the startup log I mentioned in my last email (I blame a small phone screen, yeah that is it).
I have seen a seemingly recursive drop table activity like what you are seeing however that was always after the system (7.6..04) had been up for a while and was in the process of blowing out the stack (REF KA321632). The KA mentions recursive workflow however we never were able to find any kind of filter loop. There was also a set of hotfixes for both the AR server and the Mid Tier that was supposed to address a expand menu API call that was known to send the AR stack through the roof. I asked Support about the mass drop table activity around the time our server was about to eat up its stack space (after a while I could tell by the logs when a server restart was coming due to an exhausted stack) but didn't really get a good answer as to what was performing all of the drop activity. Do you have API/Filter/SQL logging turned on as well? It would be interesting to know what API calls are happening while the server is hanging during start up. Jason On Mon, Jan 19, 2015 at 10:23 PM, BradRemedy <[email protected]> wrote: > ** > Hi Jason > > I did get SQL profiler running and checked what was being called by remedy > and found the following entries: > > SQL:BatchStarting DROP TABLE AR0int5212 182 2015-01-20 07:38:58.553 > SQL:BatchStarting DROP TABLE AR0int8000 166 2015-01-20 07:38:58.553 > SQL:BatchStarting DROP TABLE AR0int7664 178 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0float5212 182 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0float8000 166 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0float7664 178 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0char5212 182 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0char8000 166 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0char7664 178 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0decimal5212 182 2015-01-20 07:38:58.560 > SQL:BatchStarting DROP TABLE AR0decimal8000 166 2015-01-20 07:38:58.560 > Audit Logout 182 2015-01-20 07:38:58.180 > SQL:BatchStarting DROP TABLE AR0decimal7664 178 2015-01-20 07:38:58.560 > Audit Logout 166 2015-01-20 07:38:57.217 > Audit Logout 178 2015-01-20 07:38:58.020 > SQL:BatchStarting DROP TABLE AR0int7732 171 2015-01-20 07:38:58.563 > SQL:BatchStarting DROP TABLE AR0float7732 171 2015-01-20 07:38:58.567 > SQL:BatchStarting DROP TABLE AR0char7732 171 2015-01-20 07:38:58.570 > SQL:BatchStarting DROP TABLE AR0decimal7732 171 2015-01-20 07:38:58.570 > Audit Logout 171 2015-01-20 07:38:57.280 > SQL:BatchStarting DROP TABLE AR0int6804 183 2015-01-20 07:38:58.627 > SQL:BatchStarting DROP TABLE AR0float6804 183 2015-01-20 07:38:58.627 > SQL:BatchStarting DROP TABLE AR0char6804 183 2015-01-20 07:38:58.627 > SQL:BatchStarting DROP TABLE AR0decimal6804 183 2015-01-20 07:38:58.633 > > > Not sure what the system is trying to do - should i leave it and see ? > > I have also logged this with BMC Software to see if they have any ideas. > > As always, thanks for the suggestions and help guys - it is really > appreciated. I am hoping my company approves my proposal to attend the BMC > Engage event this year so that I can meet you guys and thank you in person. > > Cheers > Brad > > On Tue, Jan 20, 2015 at 6:36 AM, Jason Miller <[email protected]> > wrote: > >> ** >> >> I would definitely run a trace on the db as LJ mentioned to see what if >> any activity is going on. You can also turn on a startup log by adding a >> switch in you armonitor.cfg/config file. I can't think of it off the top of >> my head but it has been mentioned on the list before and should be on >> docs.bmc.com. >> >> Wait... I think it might be -i. It gets added to the line that actually >> starts arserver. >> >> Jason >> On Jan 19, 2015 8:28 PM, "BradRemedy" <[email protected]> wrote: >> >>> ** >>> Hi, >>> >>> I checked in the ar.cfg file and the "Record-Object-Relationship" is set >>> to "F". As a test I restarted the entire server yet the service still says >>> "Starting" with the memory usage for the arserver.exe process sitting at >>> 390mb. It will stay like this for hours and wont start. >>> >>> There is nothing in the arerror log file besides the remedy version >>> information and the startup log file has the same entries as per my >>> original post. >>> >>> The last line is still "Mon Jan 19 2015 13:40:05.7000 <Startup> <TID: >>> 0000004696> Loading field mapping information" >>> >>> Thanks in Advance >>> Brad >>> >>> On Mon, Jan 19, 2015 at 3:41 PM, Harshad Wagh <[email protected] >>> > wrote: >>> >>>> ** >>>> Hi Brad, >>>> >>>> I wonder if there is "Record Object Relationship" is set to true in >>>> under "Configuration" tab of "Server Information" tab then Remedy >>>> Application Service takes longer time to start. >>>> you can check this in ar.cfg/ar.conf file and set it to "F" and try to >>>> start the Remedy application service. >>>> >>>> >>>> >>>> Thanks >>>> Harshad >>>> >>>> -------- Original Message -------- >>>> Subject: Remedy 8.1 SP2 - Hanging on Startup >>>> From: BradRemedy <[email protected]> >>>> Date: Mon, January 19, 2015 12:30 am >>>> To: [email protected] >>>> >>>> ** >>>> Hi Guys >>>> >>>> We have a remedy 8.1 installation with dy >>>> The arstartup log file has the following entries: >>>> >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Server >>>> version = 8.1.02 201408260235* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Enabling Low >>>> Fragmentation Heap succeeded* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> InstallDir = >>>> C:\Program Files\BMC Software\ARSystem* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Set up >>>> thread control block with key = 0* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Initialize >>>> thread local storage block* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Initialize >>>> multiple-byte environment* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> >>>> LocalLanguage = ENS* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> ViewDir = en* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Initialize >>>> License library* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> >>>> LicenseFilename = C:\Program Files\Common Files\AR >>>> System\Licenses\RemedyAppServer\arsystem.lic* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Initialize >>>> Language setting and locale* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Initialize >>>> the Decimal Math library* >>>> *Mon Jan 19 2015 08:54:39.8920 <Startup> <TID: 0000007260> Open shared >>>> catalog* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Create >>>> Mutexes* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Load >>>> encryption shared library* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Load >>>> encryption static functions* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> default configuration information* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> parse environment* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> date time information* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> ApiGuidMap* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> notification strings* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> RPC queue type strings* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> filter strings* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> escalation strings* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Load system >>>> configuration file* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> >>>> arsCodeSet=windows-1252[0] arDbCodeSet=windows-1252[0]* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> pending lists* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Create cache >>>> read write lock* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Create full >>>> text status read write lock* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Create >>>> modified date field Id List lock* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> fork proxy* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> user cache* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Open log >>>> files* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Initialize >>>> XML parser* >>>> *Mon Jan 19 2015 08:54:39.9080 <Startup> <TID: 0000007260> Load AR >>>> Extensions* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Initialize >>>> process communication mechanism* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Initialize >>>> thread list* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Check >>>> multiple servers* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Initialize >>>> dead thread list* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Initialize >>>> timed calls* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> >>>> CreateRPCQueue(min=1, max=1, rpc=390600)* >>>> *Mon Jan 19 2015 08:54:39.9390 <Startup> <TID: 0000007260> Await cache >>>> ready ...* >>>> *Mon Jan 19 2015 08:54:39.9550 <Startup> <TID: 0000001020> Check >>>> licensing* >>>> *Mon Jan 19 2015 08:54:40.0010 <Startup> <TID: 0000001020> >>>> InitializeServerCache: PerformPreCachePatchDatabaseUpgrades End* >>>> *Mon Jan 19 2015 08:54:40.0010 <Startup> <TID: 0000001020> >>>> InitializeServerCache: LoadInitialSchemaInfo Begin* >>>> *Mon Jan 19 2015 08:54:40.0010 <Startup> <TID: 0000001020> Loading >>>> group list* >>>> *Mon Jan 19 2015 08:54:40.0330 <Startup> <TID: 0000001020> Initialize >>>> license pools* >>>> *Mon Jan 19 2015 08:54:40.0330 <Startup> <TID: 0000001020> Loading >>>> schema properties* >>>> *Mon Jan 19 2015 08:54:40.2670 <Startup> <TID: 0000001020> Starting >>>> preload threads* >>>> *Mon Jan 19 2015 08:54:40.2820 <Startup> <TID: 0000001020> Loading list >>>> of fields for all schemas* >>>> *Mon Jan 19 2015 08:54:44.2130 <Startup> <TID: 0000001020> Loading >>>> schema/group information* >>>> *Mon Jan 19 2015 08:54:44.4470 <Startup> <TID: 0000001020> Loading >>>> schema getlist/index/archive/audit information* >>>> *Mon Jan 19 2015 08:54:45.1490 <Startup> <TID: 0000001020> Loading >>>> schema/vui information* >>>> *Mon Jan 19 2015 08:54:45.3370 <Startup> <TID: 0000001020> Loading >>>> field mapping information* >>>> >>>> >>>> ************************************************************************* >>>> >>>> Anyone had this before? Appears that the field mapping information is >>>> corrupt or running into a problem and I am not sure how best to move >>>> forward with this. >>>> >>>> Any ideas would be appreciated >>>> >>>> Cheers >>>> Brad >>>> >>>> >>>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>>> >>>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>>> >>> >>> _ARSlist: "Where the Answers Are" and have been for 20 years_ >>> >> _ARSlist: "Where the Answers Are" and have been for 20 years_ >> > > _ARSlist: "Where the Answers Are" and have been for 20 years_ > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
