You may want to compare the data in the system between your test and prod envs. I'm not a DBA, but what I've seen seems to be that the more rows in your db that need to have columns added / changed / etc, the longer it takes.
----- Original Message ----- From: "Jason Miller" <jason.mil...@gmail.com> To: arslist@ARSLIST.ORG Sent: Thursday, October 20, 2011 6:01:43 PM Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** The hardware size appears to have little to do with the install times. Of course there is some type of relation between hardware and time but I can't figure it out. In our case the production db is a physical server with 24 cores and 48 gb of ram and test is a VM with 4 CPUs assigned and 8 gb of ram. The production app server is a VM with 2 cores and 6 gb of ram and test is a VM with 2 cores and 4 gb of ram. There is a little more data in the production db vs. the test db. I would think this would fairly negligible since both were not really being used. I scheduled our production outage for 48 hours (with a little extra time) based on the time it took to perform the same steps in test with the additional step of running BPCU (I skipped this in production since we had about 6 custom/modified objects that I converted manually). Test took a little less than 48 hours and this included ironing out the process and had the extra BPCU time. I was pretty certain upgrading production would be much faster. It turns out that it took production longer than a lighter weight test environment. Jason On Thu, Oct 20, 2011 at 1:09 PM, strauss < stra...@unt.edu > wrote: ** I restored my test/sample data system back to where ITSM was still SP1 (ARS and Atrium at SP2) and performed two manual interventions before kicking off ITSM 7.6.04 SP2 again: Delete join form VIS:Status_Stage_Flow_TransitionRulesLookup which is carrying an outdated View ID for the Default Administrator View (the only view present) I am told that this was _ supposed _ to have been changed in the upgrade to 7.6.03, but none of my systems reflect that change. Delete all existing records in DLD:SYS:DataLoadStatus that were preventing SP2 from creating a unique index on InstanceId There were 126 old records in there with GUIDs that are NOT UNIQUE on a case-insensitive system (SQL Server). Looking at the production system which had no records in that form, it actually looks like the unique index was created by the upgrade to 7.6.04.0000 – which is when I have been told it was introduced. Somehow, my test system got past that and SP1 both without the installation hanging – until SP2. The installation of ITSM 7.6.04 SP2 has been running since 10:30 AM; we’ll see how long it takes, but this is on a low powered VM and isn’t a good measure of what it will take on production grade equipment. From what you’re saying, Pat, it doesn’t do very well on MUCH higher-powered servers, either. I guess the Thanksgiving holiday will be my first window of opportunity for applying a service pack that runs for hours and hours and hours to the production environment. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of patrick zandi Sent: Thursday, October 20, 2011 2:49 PM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** Yeah::: Upgrades look like they are going to disappear for the create a new server and migrate data option soon. Most change windows: especially Enterprise wise:: could they really go down for 5-7 hours? NO! not happening.. so My guess is there is some sort of single FIFO job process single Java thread into the server needs to be change to something a little more robust. don't know what the answer is.. but I can see this as an issue... I have 64 CPU's and it took forever.. cpu never went over 6%... had tons of ram too... not sure what the answer is ? BMC Engineering might have a better clue... On Thu, Oct 20, 2011 at 3:25 PM, Jason Miller < jason.mil...@gmail.com > wrote: ** Congratulations! Could somebody from BMC explain what is happening during this time where nothing appears to be happening? I remember having the same experience during the 7.5 to 7.6.04 SP1 upgrade. The logs do not indicate that anything is happening and a db trace shows very little activity. You just have to hold your breath, have faith and hope for the best. Below is a snippet of our production upgrade notes the times of each install. Fortunately I was able to take the system down for two days because we only had three people using this new system for production data (contracts). We finished just a hair past our 9 AM time to return to production (I got lucky). ++++++++++++++++++++++++++++++++++++ Run ARSystem_7604SP1_Windows (5/25/2011 14:02 - 16:00) • A handful of errors: § arerror.log § Wed May 25 15:01:05 2011 390600 : Could not find or read the tags file. (ARNOTE 452) § Wed May 25 15:01:05 2011 C:\Program Files\Common Files\AR System\Licenses\<servername>\arsystem.tag § Wed May 25 15:02:03 2011 390600 : The value(s) for this entry violate a unique index that has been defined for this form (ARERR 382) § Wed May 25 15:02:02 2011 390600 : Creation of an application form failed. (ARERR 4550) § An error regarding the email engine. Run atrium7604SP1.win (5/25/2011 23:17 - 5/26/2011 13:21) • RIK hung at 00:23, acknowledged and process started again at 08:03 • Finished without further error • The issues in test with Active Link guides after Atrium Core did not happen in production • Create Windows symbolic link to account for AIE directory change § mklink /J "D:\Program Files (x86)\BMC Software\AtriumCore\aie\service" "D:\Program Files (x86)\BMC Software\AtriumCore\aie\service64" Run BMCRemedyITSMSuite7604SP1.windows (5/26/2011 14:00 - 5/27/2011 07:33) Run ServiceLevelManagement7.6.04.SP1.windows (5/27/2011 08:07 - 09:02) Run BmcRkm7604SP1.windows (5/29/2001 23:30 - 5/30/2011 02:04) Run atriumintegrator7604SP1.client_server.win / atriumintegrator7604.server.win (6/4/2011 20:56 - 21:30) ++++++++++++++++++++++++++++++++++++ Jason On Thu, Oct 20, 2011 at 5:38 AM, patrick zandi < remedy...@gmail.com > wrote: ** ok, it finished: I ran it over night// had several warnings.. but no failures.. off to the logs.. total time :: 7.5 hours On Wed, Oct 19, 2011 at 3:09 PM, strauss < stra...@unt.edu > wrote: ** My installs are failing after only 70 or 80 minutes. I cleaned up the first defect and it has now failed on another. Apparently SP2 depends on things that failed to happen properly during my 7.6.03 upgrades from earlier versions, so it is unable to complete. Nothing here to instill confidence for planning a rapid move to SP2 in the production environment. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of patrick zandi Sent: Wednesday, October 19, 2011 1:56 PM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** OK back at it again::: According to Support::: so now I am asking again::: ENV: 64 cpu's 32 gig of ram How many hours is the task update suppose to take? 6 - 8 - 12 - 24 hours? Cause were over 5 already.. On Tue, Oct 18, 2011 at 6:11 PM, Patrick Zandi < remedy...@gmail.com > wrote: So they broke it twice ! Lol Sent from my iPhone so typo's or funky words can and do happen! On Oct 18, 2011, at 6:02 PM, strauss < stra...@unt.edu > wrote: ** BMC Support indicates that there is a known defect with a view ID that basically prevents the ITSM SP2 installer from completing successfully. It looks like it will affect all of my available environments. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of strauss Sent: Tuesday, October 18, 2011 1:43 PM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** The rik.err error was: Definition File (D:\Program Files (x86)\BMC Software\BMCRemedyITSMSuite\ITSMT01\Workflow\install\7.6.04_sp2_overlay_install\systems\fnd\workflow\en\.\fnd_deplapp.def) import failed with error code (392) :- (Field/VUI name must be unique for the form -- there is already a field or VUI using this name) Since there are almost 300 forms in that def file, I’m going to let BMC figured out what they did wrong. The only form I overlaid on the server is HPD:Help Desk (one field, Reported Source was given Other as a Default value) and one of the 7.6.04 upgrades added an overlay for HPD:HelpDesk_AuditLogSystem, and neither of those is in the problematic def file. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of strauss Sent: Tuesday, October 18, 2011 1:19 PM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** Oh, I doubt that (that there is any hope for me)… my downloads are from last week. The Atrium SP2 applied clean on the second try, but the ITSM SP2 just completed with a failure. So far the SP2 installers are NOT looking good at all. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of patrick zandi Sent: Tuesday, October 18, 2011 1:11 PM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** you will not believe it:: LIKE THIS HAS NEVER HAPPENED:::: Files uploaded to the download site are BAD::: they just replaced them with the correct files yesterday?!? I guess.. Go Figure.. So I have re reinstall everything Again.. Wonderful.. So there is hope for ya Chris!!! LOL........ On Tue, Oct 18, 2011 at 1:29 PM, Grooms, Frederick W < frederick.w.gro...@xo.com > wrote: ** Do you have an SQL log?? It looks like it is updating 1 record at a time Fred From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of patrick zandi Sent: Tuesday, October 18, 2011 11:57 AM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** OK, 3 hours of this::: NOW WHAT>>>???? (Oct 18 2011 03:54:07.112 PM +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, LOG EVENT {Description=[Code],Detail=[550]} (Oct 18 2011 03:54:07.113 PM +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, LOG EVENT {Description=[Code],Detail=[551]} (Oct 18 2011 03:54:07.113 PM +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, LOG EVENT {Description=[Code],Detail=[552]} (Oct 18 2011 03:54:07.114 PM +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, LOG EVENT {Description=[Code],Detail=[8303]} (Oct 18 2011 03:54:07.115 PM +0000),INFO,com.bmc.install.product.rik.statement.RIKLoadApplicationStatement, LOG EVENT {Description=[RIK load application command],Detail=[/var/tmp/Utilities/rik/rik loadapp -f /u01/app/ITSM/Workflow/install/7.6.04_sp2_overlay_install/systems/fnd/workflow/en/adf_fnd.xml -m nosamp -x HOSTNAME -t 2106 -u Demo -p ** -l /u01/app/ITSM -n InstallRAF7604 -L]} On Tue, Oct 18, 2011 at 12:12 PM, patrick zandi < remedy...@gmail.com > wrote: OK, Atrium core sp2 was not a problem, but I was doing oob install too. the bmc itsm overlay install is running now, for over 2 hours... I am getting a little concerned.. it appears stuck on task progress, executing.. I do not see anything in the /var/tmp files .. I must be missing something.. but we shall see.. On Tue, Oct 18, 2011 at 10:14 AM, strauss < stra...@unt.edu > wrote: ** Sounds like you had better luck than I. I started running the SP2 installs on my sample data environment which has been stable on some lightweight 2008 R2 VMs since 7.5, and was upgraded to 7.6, 7.6.03, 7.6.04, and SP1 without incident. The SP2 ARS installer ran just fine – updated ALL components. The Atrium installer exploded in a shower of sparks and left the system as toast; apparently it tried unsuccessfully to restart the AR Service during the install, and stopped the service but not the process – which remained connected to the SQL Server, and proceeded to spew some of the coolest error messages I have seen in a while; here is a selection for your entertainment: Mon Oct 17 15:15:26 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 15:15:26 2011 ITSMT01 (9556) BMC.FILTERAPI.NOE : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 15:15:45 2011 : Action Request System(R) Server x64 Version 7.6.04 SP2 201110080614 (c) Copyright 1991-2011 BMC Software, Inc. Mon Oct 17 15:15:45 2011 ProcessMain : Another copy of the server is already running on the same RPC socket (ARERR 35) Mon Oct 17 15:15:45 2011 d:*program files (x86)*bmc software*arsystem*confRemedyMutex390600 Mon Oct 17 15:15:45 2011: AR System server terminated — fatal error occurred in ARSERVER (ARNOTE 21) Mon Oct 17 16:04:22 2011 390602 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:04:22 2011 ITSMT01 (9999) RMDY.ITSM.RKM.FILESYSTEM : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:04:23 2011 390602 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:04:26 2011 ITSMT01 (22701) REMEDY.ARDBC.SERVER.ADMINISTRATION : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:06:19 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:06:19 2011 ITSMT01 (22701) : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:06:20 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:06:20 2011 itsmt01.unt.edu (9999) ARSYS.ARF.REGISTRY : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:06:21 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:06:21 2011 ITSMT01 (9556) BMC.FILTERAPI.NOE : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:06:22 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:06:22 2011 ITSMT01 (9999) RMDY.ITSM.RLE : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:06:23 2011 390600 : Cannot establish a network connection to the AR System Plug-In server (ARERR 8760) Mon Oct 17 16:06:23 2011 itsmt01.unt.edu (9998) ARSYS.ARF.FTS : RPC: Miscellaneous tli error - System error (Connection refused) Mon Oct 17 16:18:06 2011 : Action Request System(R) Server x64 Version 7.6.04 SP2 201110080614 (c) Copyright 1991-2011 BMC Software, Inc. Mon Oct 17 16:18:07 2011 ProcessMain : Another copy of the server is already running on the same RPC socket (ARERR 35) Mon Oct 17 16:18:07 2011 d:*program files (x86)*bmc software*arsystem*confRemedyMutex390600 Mon Oct 17 16:18:07 2011: AR System server terminated — fatal error occurred in ARSERVER (ARNOTE 21) So far I am not impressed. Also, the problem with logging in that I have been fighting in Migrator 7.6.04 – all versions since 7.6.03 – was not fixed in SP2, so that was no improvement. Christopher Strauss, Ph.D. Call Tracking Administration Manager University of North Texas Computing & IT Center http://itsm.unt.edu/ From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG ] On Behalf Of patrick zandi Sent: Tuesday, October 18, 2011 8:14 AM To: arslist@ARSLIST.ORG Subject: Re: ARS 7.6.04 SP2 Solaris installer Oracle db. ** This was an upgrade from 7.6.04 SP1 -- forgot to mention. On Tue, Oct 18, 2011 at 9:05 AM, patrick zandi < remedy...@gmail.com > wrote: OK folks, ARS 7.6.04 SP2 (released Oct 11) - Oracle 11.iR2 Solaris 10 I installed, and the installer was pretty intelligent, and I had 1 minor weird issue, and the fix was odd. Where the page is that asked for the aradmin and password (put in the password). mine was default for aradmin however it did not like it when I got to the page that asked for the system password it says the aradmin password was wrong (you would think that was already done).. so if you go back 4 - 5 steps and put in the aradmin and password, then as you move forward : bang it knows the rest and does it (fairly intelligent). Strange but it went through the install without a hitch.. I have not see anything like this in a while.. have not tested fully.. but wanted you all to know.. we are making progress... Thanks BMC -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ -- Patrick Zandi _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG11 www.wwrug.com ARSlist: "Where the Answers Are"_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
