Out of 24 AR Servers, one will be primary for which following steps will have to be done For remainder 23 which are secondary servers with 9.1.03 parallel upgrades are supported, under 30 mins platform should get upgraded. Also none of the 23 servers will hit the problem mentioned below because in case of secondary DB updates don't need to be done
Regards, Abhijeet From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Brian Pancia Sent: 11 October 2017 00:22 To: arslist@ARSLIST.ORG Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** Wow. 24 ARSystem servers. That is impressive. All having the same issue too with the installers. Good luck with that. ________________________________ From: Action Request System discussion list(ARSList) <arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG>> on behalf of Gadgil, Abhijeet <abhijeet_gad...@bmc.com<mailto:abhijeet_gad...@bmc.com>> Sent: Tuesday, October 10, 2017 1:03:33 PM To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** Looked the logs that were mentioned at the start of this email thread. We see a couple of issues within the logs and calling these out in reverse chronological order: 1. SEVERE error regarding server not starting up - this is a result of database changes themselves not going thru correctly in the first place. 2. While making database changes, there is a ORA-02437 error a. (Oct 06 2017 02:52:22.617 PM +0200),WARNING,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerManageUpgradeJDBCWriterErrorHandler, LOG EVENT {Description=[Failed to create primary key when running sql: ALTER TABLE ARADMIN.FILTER_PUSH ADD PRIMARY KEY ( filterId, actionIndex, fieldId, overlayGroup ). Upgrade will continue, but upon completion database should be manually corrected to ensure product performs as designed.],Detail=[ORA-02437: cannot validate (ARADMIN.SYS_C0062062) - primary key violated 3. Even with just this error, we would expect only some of the imports for FILTERs to fail. But, there are a lot many failures related to duplicate inserts. Looking at the log in detail, we see that database changes didn't go through correctly. This is evident from the log of ARServerPriorToServerJDatabaseUpgradeTask class, really missing logs from this class tell us this class didn't go through and inserts are failing with index violation error. Why did this class not execute - really either of the 2 conditions: a. Either the server being upgraded is secondary server - this is not the case and it's confirmed in the logs. b. Or it did not find "featureARServer" product ID and version in ARSystemInstalledConfiguration.xml that existed in the install directory before starting the upgrade - this seems to be true in this case as Thomas mentioned in the mail below "ARSystemInstalledConfiguration.xml didn't exist but it was created after the first failed installation. In there it says, server is NOT server group member. So does ar.conf". We have seen similar issue in the past due to corrupt/incomplete or missing ARSystemInstalledConfiguration.xml file. Now why this file does not exist before upgrade is the real root cause, which you can check based on audit enabled on OS. Fix for this issue: ? Revert the system completely back to 8.1 clean state including database and file system. ? Ensure file system has correct ARSystemInstalledConfiguration.xml file with lines such as this: o <productFeature backupOnUpgrade="false" id="featureARServer" independentOfChildren="true" parent="featureARSystemServers" rebootRequiredOnInstall="false" rebootRequiredOnUninstall="false" rebootRequiredOnUpgrade="false" requiredDiskSpaceMode="default.windows" state="INSTALLED" visible="true"> <version majorVersion="1" minorVersion="00" releaseVersion="8"/> ... ? Ensure there are no duplicates in FILTER_PUSH table. Use this query to find if there are duplicates before upgrade: o select filterId, actionIndex, fieldId, count(*) from filter_push group by filterId, actionIndex, fieldId having count(*) > 1 o If above query finds duplicates, that needs to be fixed before starting the upgrade. Clearly there seem to be corrupt entries related to overlays of those filters. Don't see how it will happen via the product. Then start upgrade with 9.1.03 and this time it should succeed once database is upgraded successfully. Regards, Abhijeet From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Carl Wilson Sent: 10 October 2017 22:19 To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** Hi, As BMC have had many issues with the 9.x installers (e.g. patches for patch installers), have you tried a based upgrade to 9.0/9.1 then done the 9.1 SP? Might be worth a try. ---------------------------------------------- Kind Regards, Carl Wilson From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Thomas Miskiewicz Sent: 10 October 2017 16:46 To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** Carl, the pre-check tool is totally bonkers. It's reposting misleading and false errors and does randomly so. From my experience it cannot be taken seriously in 95% cases. This time it reported this: Filter Correction Check Description In BMC Remedy AR System 8.1.02, the Ap:Alt-SetPermissions filter was introduced in one of the hotfixes (HF\pinatubo\CertifiedHotFixes\8.1.02\Multitenancy\SW00488367-ApprovalServer\MultiTenancy_Approval_Hotfix_81SP2). The same filter in 9.1.02 is included with an upper case 'P' in the form name: AP:Alt-SetPermissions. When upgrading from 8.1.02 with that hotfix to 9.1.02, you encounter the following error: "ERROR RIKMain - 382 The value(s) for this entry violate a unique index that has been defined for this form" The check validates the casing in the AP:Alt-SetPermissions filter name. Type Pre-upgrade check Check performed and expected result When upgrading from 8.1.02 with that hotfix to 9.1.02, the check detects a lower case 'p' in the AP:Alt-SetPermissions filter name and displays the following error message: Please correct the filter name from Ap:Alt-SetPermissions to AP:Alt-SetPermissions. Execute the following SQL statement to fix it: UPDATE filter SET name = 'AP:Alt-SetPermissions', resolvedName = 'AP:Alt-SetPermissions' WHERE name = 'Ap:Alt-SetPermissions'; Corrective action Execute the SQL statement provided in the preceding row to update the filter name. I ran this statement and of course none records were modified. So why bother? Thomas On 10. Oct 2017, at 17:36, Carl Wilson <carlbwil...@gmail.com<mailto:carlbwil...@gmail.com>> wrote: ** Hi, As Brian has mentioned, you could do the first server as a staged upgrade then for the other servers do a binary update on the remaining servers (as the DB will be already updated) .... It is really getting the DB information updated with the first run, then the binaries on the other 23 servers. Not surprising you are having issues, BMC seem to forget about people in the rest of the world when they create their installers and think that everyone runs "US English" as a default. BTW: Your install log mentions that the pre-checker failed, but there is no reference to what failed in the pre-check. What is in the report for this ('file:///ESP/usr/ar/ARSE1/Logs/result/configchecker_report_1507293439188.html')? ---------------------------------------------- Kind Regards, Carl Wilson From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Thomas Miskiewicz Sent: 10 October 2017 15:55 To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** I didn't try and I won't. I have 24 servers to upgrade!!! On 10. Oct 2017, at 16:51, Brian Pancia <panc...@finityit.com<mailto:panc...@finityit.com>> wrote: ** Have you tried to upgrade one sp/version at a time instead of going directly to 9.1.03. I've done countless upgrades and never had a lot of luck with going from one version (ie 8.x to 9.x). In that case I've always did a clean install and migrate data. RRRChive does an excellent job at moving data. I've seen installers go through stating successful only to find out things were missing on the back end. I would recommend trying an incremental upgrade first and if you're spinning your tires do a clean install and migrate data. It might be quicker then troubleshooting the install. Brian ________________________________ From: Action Request System discussion list(ARSList) <arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG>> on behalf of Thomas Miskiewicz <tmisk...@gmail.com<mailto:tmisk...@gmail.com>> Sent: Tuesday, October 10, 2017 8:14:40 AM To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed ** No, the issue is that the installer updates the binaries and then cannot start the server and gives up. I also feel like giving up after the BMC support cannot tell us what their product does. Thomas On 10. Oct 2017, at 14:08, Saji Philip <sphili...@gmail.com<mailto:sphili...@gmail.com>> wrote: ** Is this an upgrade in place? Does the installer hang (never finishes and no errors). We had a similar issue and the 9..x would fail ARS because we had a custom AL guide and it would error out on tha arcontainer table. Taking a backup and deleting this AL guide allowed ARS to install. Then we had problems with Atrium in that it didn't like our overlay on the BMC.Elements form. But that would error out. We just deleted the overlay. But the weirdest part is that ITSM would hang in installation. Putting the server to Allow queries to be true, even though the prechecker said otherwise, allowed the ITSM to go through... 8.1.02 going to 9.1.03 On Tue, Oct 10, 2017, 6:27 AM Thomas Miskiewicz <tmisk...@gmail.com<mailto:tmisk...@gmail.com>> wrote: ** Satya, the environment is NOT a copy or clone of another environment. The servgrp... Tables don't exist. I did a clean start with deleting logs etc few times already. Nothing new. Same stuff in the logs. I would appreciate if you could review my armonitor.conf and ARSystemInstalledConfiguration.xml Maybe you see something's I don't. Apart from that I have no idea what else I can do. Even worse neither does BMC! They've been discussing the matter internally since a couple of days but didn't approach us with a single suggestion. When this happens (and this is how BMC support in India works 99.9% of time), they usually wait two weeks and send me a question whether the problem still persists. How ridiculous is that! Thomas On 6. Oct 2017, at 20:05, Satya Gandhi <satya.gan...@gmail.com<mailto:satya.gan...@gmail.com>> wrote: ** Hi Thomas, A dbVersion value of 54 indicates that the database has been upgraded I also see that there is alternating information about the ARServer being secondary serher AT least when it is validating FTS ports If the database was a copy from another environment which was part of a server group, then perform the flowing steps Remove the old arerror log and armonitor log files and restart the arsystem services. If the reserves starts successfully the perform the following steps Delete records from table servgrp_board and servgrp_resources Change db version value to what is currently on prevdbversion column and remove the value from prevdbversion On the AR config file, set multiple ar servers to false and server group member to false. Also ensure that there are no references for secondary server for any plugins or any features. Comment those lines out, if you find any Remove any files from the temp directory to have clean set of install log files Ensure that the arsystem server licenses are valid and you are able to create new records in any for that already have more than 2000 entries Also set the min and max values for private queues, fast and list threads to the current and recommended maximum value for your infrastructure Now run the installer and let us know how it goes Thanks Satya Gandhi On 6 Oct 2017 6:29 pm, "Thomas Miskiewicz" <tmisk...@gmail.com<mailto:tmisk...@gmail.com>> wrote: ** Hi, the dbVersion is 54. Server is not part of the server group, still the stupid installer says: server is NOT server group member, server IS server group member, server is NOT server group member ... why? (Oct 06 2017 02:29:12.122 PM +0200),CONFIG,com.bmc.install.task.InstallationPropertiesHelper, LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_GROUP_MEMBER],Detail=[F]} [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Num-Preload-Schema-Segments, , 25, searchKEYandREPLACEvalue]], [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Num-Preload-Threads, , 10, searchKEYandREPLACEvalue]], [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Application-Enable, F, T, setCFGentry]], [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Large-Result-Logging-Threshold, , 1000000, setCFGentry]], [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Max-Log-History, , 8, setCFGentry]], [className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Max-Log-File-Size, , 134217728, setCFGentry]]]], [level=Database Only Server Group Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Server-Group-Member, , F, searchKEYandREPLACEvalue]]]], [level=AR Server Group Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Server-Group-Member, , T, searchKEYandREPLACEvalue]]]], [level=Configure JMX,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Jmx-port, , 61500, searchKEYandREPLACEvalue]]]], [level=Configure JMS,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Default-messaging-port, , 61617, searchKEYandREPLACEvalue]]]], [level=AR Server Atrium SSO Configurati (Oct 06 2017 02:47:07.676 PM +0200),INFO,com.bmc.smbu.install.common.rule.engine.StageGroup, LOG EVENT {Description=[Skipping execution of stage],Detail=[[level=Database Only Server Group Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Server-Group-Member, , F, searchKEYandREPLACEvalue]]]]]} (Oct 06 2017 02:47:07.676 PM +0200),INFO,com.bmc.smbu.install.common.rule.engine.Stage, LOG EVENT {Description=[Executing stage [level=AR Server Group Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf, Server-Group-Member, , T, searchKEYandREPLACEvalue]]]]]} (Oct 06 2017 02:47:07.676 PM +0200),INFO,com.bmc.smbu.install.common.rule.engine.extension.ExtensionCommand, LOG EVENT {Description=[Executing extension Command]} etc. arsystem_install_log.txt attached ARSystemInstalledConfiguration.xml didn't exist but it was created after the first failed installation. In there is say, server is NOT server group member. So does ar.conf I'm on Solaris in this case. arerror.log has a few hundreds entries complaining that he couldn't import forms: Fr Okt 06 15:26:28.836 2017 Import failed for AR System Administration: Add Or Remove Licenses: Fr Okt 06 15:26:28.953 2017 Import failed for AR System Administration: Display Form To Collect User Decisions: Fr Okt 06 15:26:29.225 2017 Import failed for AR System License: Save Produse Attachment: Fr Okt 06 15:26:29.530 2017 Import failed for AR System Licenses Console: Fr Okt 06 15:26:29.688 2017 Import failed for AR System: Generate License Usage Report: Fr Okt 06 15:26:29.986 2017 Import failed for AR System Licenses Audit: Fr Okt 06 15:26:30.229 2017 Error updating audit shadow form AR System Licenses Audit Thomas _ARSlist: "Where the Answers Are" and have been for 20 years_ ** On 6. Oct 2017, at 19:13, Satya Gandhi <satya.gan...@gmail.com<mailto:satya.gan...@gmail.com>> wrote: ** Hi, Check the record in control table from a db utility Select * from control The value in dbVersion column should be 54 and if that's not the case, the database has not been upgraded to 9.1.03. If the AR Server binaries has been upgraded but not the database, t It should given an error Bout column useShaH (or something similar). This column is created only when the database has also been upgraded Was the server part of server group? If yes, can you please check if the server was considered a secondary server and hence updates only the binaries and not the database. This can be found on the systeminstalledconfiguration.xml file If you are running the server on Windows, check the productregistry.xml to find the install and upgrade entries for product BMC Remedy action request system server I cannot open the log files as I am on my mobile right now. Are there any glaring errors on arsystem install logs? Regards Satya Gandhi On 6 Oct 2017 18:06, "Grooms, Frederick W" <frederick.w.gro...@xo.com<mailto:frederick.w.gro...@xo.com>> wrote: Is there anything in the arDebug.log or arError.log? -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG>] On Behalf Of Thomas Miskiewicz Sent: Friday, October 06, 2017 11:24 AM To: arslist@ARSLIST.ORG<mailto:arslist@ARSLIST.ORG> Subject: Second try --> Upgrade 8.1.00 to 9.1.03 Failed Hello Listers, at the moment I do ARS upgrades from 7.6.4, 8.1.00 and higher to 9.1.03 for various customers. OS: Linux and Solaris. Not a single system that just worked! I'm really not sure how to move to production under this circumstances... What's even worse: the BMC support in India has zero clue and just keeping us busy with their ideas and requests. In all the month not a single hint with substance! BTW, talked to some partners and customers which I do not work for - same issues! But BMC support always pretends they never heard of any issues. The logs of my recent installation:https://s3.eu-central-1.amazonaws.com/miskiewicz/arslist/INC112333.tar.gz Maybe one of you has an idea what is going on. Basically the job went like this: 1. Clean and working 8.1.00 installation. 2. Installer 9.1.03 runs. 3. Installer 9.1.03 cannot start the server it installed and gives up. 4. Result: A messed up ARS 9.1.03 installation that doesn't start, but is trying to do a gazzilion of inserts into various tables and is violating the unique index at the same time. I restored and started from scratch. Same result. Not sure what to do. The only comfort is BMC doesn't know either. I feel less stupid. Thomas _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org<http://www.arslist.org/> "Where the Answers Are, and have been for 20 years" _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ DISCLAIMER: The information contained in this e-mail and its attachments contain confidential information belonging to the sender, which is legally privileged. The information is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are notified that any disclosure, copying, distribution or action in reliance upon the contents of the information transmitted is strictly prohibited. If you have received this information in error, please delete it immediately. _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ _ARSlist: "Where the Answers Are" and have been for 20 years_ DISCLAIMER: The information contained in this e-mail and its attachments contain confidential information belonging to the sender, which is legally privileged. The information is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are notified that any disclosure, copying, distribution or action in reliance upon the contents of the information transmitted is strictly prohibited. If you have received this information in error, please delete it immediately. _ARSlist: "Where the Answers Are" and have been for 20 years_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
