Looked the logs that were mentioned at the start of this email thread.
We see a couple of issues within the logs and calling these out in reverse
chronological order:
1. SEVERE error regarding server not starting up – this is a result of
database changes themselves not going thru correctly in the first place.
2. While making database changes, there is a ORA-02437 error
a. (Oct 06 2017 02:52:22.617 PM
+0200),WARNING,com.bmc.install.product.arsuitekit.platforms.arsystemservers.arserver.ARServerManageUpgradeJDBCWriterErrorHandler,
LOG EVENT {Description=[Failed to create primary key when running sql: ALTER
TABLE ARADMIN.FILTER_PUSH ADD PRIMARY KEY ( filterId, actionIndex, fieldId,
overlayGroup ). Upgrade will continue, but upon completion database should be
manually corrected to ensure product performs as designed.],Detail=[ORA-02437:
cannot validate (ARADMIN.SYS_C0062062) - primary key violated
3. Even with just this error, we would expect only some of the imports
for FILTERs to fail. But, there are a lot many failures related to duplicate
inserts.
Looking at the log in detail, we see that database changes didn’t go through
correctly. This is evident from the log of
ARServerPriorToServerJDatabaseUpgradeTask class, really missing logs from this
class tell us this class didn’t go through and inserts are failing with index
violation error.
Why did this class not execute – really either of the 2 conditions:
a. Either the server being upgraded is secondary server – this is not the
case and it’s confirmed in the logs.
b. Or it did not find “featureARServer” product ID and version in
ARSystemInstalledConfiguration.xml that existed in the install directory before
starting the upgrade – this seems to be true in this case as Thomas mentioned
in the mail below “ARSystemInstalledConfiguration.xml didn’t exist but it was
created after the first failed installation. In there it says, server is NOT
server group member. So does ar.conf”.
We have seen similar issue in the past due to corrupt/incomplete or missing
ARSystemInstalledConfiguration.xml file.
Now why this file does not exist before upgrade is the real root cause, which
you can check based on audit enabled on OS.
Fix for this issue:
· Revert the system completely back to 8.1 clean state including
database and file system.
· Ensure file system has correct ARSystemInstalledConfiguration.xml
file with lines such as this:
o <productFeature backupOnUpgrade="false" id="featureARServer"
independentOfChildren="true" parent="featureARSystemServers"
rebootRequiredOnInstall="false" rebootRequiredOnUninstall="false"
rebootRequiredOnUpgrade="false" requiredDiskSpaceMode="default.windows"
state="INSTALLED" visible="true">
<version majorVersion="1" minorVersion="00" releaseVersion="8"/>
…
· Ensure there are no duplicates in FILTER_PUSH table. Use this query
to find if there are duplicates before upgrade:
o select filterId, actionIndex, fieldId, count(*) from filter_push group by
filterId, actionIndex, fieldId having count(*) > 1
o If above query finds duplicates, that needs to be fixed before starting the
upgrade. Clearly there seem to be corrupt entries related to overlays of those
filters. Don’t see how it will happen via the product.
Then start upgrade with 9.1.03 and this time it should succeed once database is
upgraded successfully.
Regards,
Abhijeet
From: Action Request System discussion list(ARSList)
[mailto:[email protected]] On Behalf Of Carl Wilson
Sent: 10 October 2017 22:19
To: [email protected]
Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed
**
Hi,
As BMC have had many issues with the 9.x installers (e.g. patches for patch
installers), have you tried a based upgrade to 9.0/9.1 then done the 9.1 SP?
Might be worth a try.
----------------------------------------------
Kind Regards,
Carl Wilson
From: Action Request System discussion list(ARSList)
[mailto:[email protected]] On Behalf Of Thomas Miskiewicz
Sent: 10 October 2017 16:46
To: [email protected]<mailto:[email protected]>
Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed
**
Carl,
the pre-check tool is totally bonkers. It’s reposting misleading and false
errors and does randomly so. From my experience it cannot be taken seriously in
95% cases.
This time it reported this:
Filter Correction Check
Description
In BMC Remedy AR System 8.1.02, the Ap:Alt-SetPermissions filter was introduced
in one of the hotfixes
(HF\pinatubo\CertifiedHotFixes\8.1.02\Multitenancy\SW00488367-ApprovalServer\MultiTenancy_Approval_Hotfix_81SP2).
The same filter in 9.1.02 is included with an upper case 'P' in the form name:
AP:Alt-SetPermissions. When upgrading from 8.1.02 with that hotfix to 9.1.02,
you encounter the following error:
"ERROR RIKMain - 382 The value(s) for this entry violate a unique index that
has been defined for this form"
The check validates the casing in the AP:Alt-SetPermissions filter name.
Type
Pre-upgrade check
Check performed and expected result
When upgrading from 8.1.02 with that hotfix to 9.1.02, the check detects a
lower case 'p' in the AP:Alt-SetPermissions filter name and displays the
following error message:
Please correct the filter name from Ap:Alt-SetPermissions to
AP:Alt-SetPermissions. Execute the following SQL statement to fix it:
UPDATE filter SET name = 'AP:Alt-SetPermissions', resolvedName =
'AP:Alt-SetPermissions' WHERE name = 'Ap:Alt-SetPermissions';
Corrective action
Execute the SQL statement provided in the preceding row to update the filter
name.
I ran this statement and of course none records were modified. So why bother?
Thomas
On 10. Oct 2017, at 17:36, Carl Wilson
<[email protected]<mailto:[email protected]>> wrote:
**
Hi,
As Brian has mentioned, you could do the first server as a staged upgrade then
for the other servers do a binary update on the remaining servers (as the DB
will be already updated) …. It is really getting the DB information updated
with the first run, then the binaries on the other 23 servers.
Not surprising you are having issues, BMC seem to forget about people in the
rest of the world when they create their installers and think that everyone
runs "US English" as a default.
BTW: Your install log mentions that the pre-checker failed, but there is no
reference to what failed in the pre-check. What is in the report for this
('file:///ESP/usr/ar/ARSE1/Logs/result/configchecker_report_1507293439188.html')?
----------------------------------------------
Kind Regards,
Carl Wilson
From: Action Request System discussion list(ARSList)
[mailto:[email protected]] On Behalf Of Thomas Miskiewicz
Sent: 10 October 2017 15:55
To: [email protected]<mailto:[email protected]>
Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed
**
I didn’t try and I won’t. I have 24 servers to upgrade!!!
On 10. Oct 2017, at 16:51, Brian Pancia
<[email protected]<mailto:[email protected]>> wrote:
**
Have you tried to upgrade one sp/version at a time instead of going directly to
9.1.03. I've done countless upgrades and never had a lot of luck with going
from one version (ie 8.x to 9.x). In that case I've always did a clean install
and migrate data. RRRChive does an excellent job at moving data. I've seen
installers go through stating successful only to find out things were missing
on the back end. I would recommend trying an incremental upgrade first and if
you're spinning your tires do a clean install and migrate data. It might be
quicker then troubleshooting the install.
Brian
________________________________
From: Action Request System discussion list(ARSList)
<[email protected]<mailto:[email protected]>> on behalf of Thomas
Miskiewicz <[email protected]<mailto:[email protected]>>
Sent: Tuesday, October 10, 2017 8:14:40 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: Second try --> Upgrade 8.1.00 to 9.1.03 Failed
**
No, the issue is that the installer updates the binaries and then cannot start
the server and gives up. I also feel like giving up after the BMC support
cannot tell us what their product does.
Thomas
On 10. Oct 2017, at 14:08, Saji Philip
<[email protected]<mailto:[email protected]>> wrote:
**
Is this an upgrade in place? Does the installer hang (never finishes and no
errors). We had a similar issue and the 9..x would fail ARS because we had a
custom AL guide and it would error out on tha arcontainer table. Taking a
backup and deleting this AL guide allowed ARS to install. Then we had problems
with Atrium in that it didn't like our overlay on the BMC.Elements form. But
that would error out. We just deleted the overlay. But the weirdest part is
that ITSM would hang in installation. Putting the server to Allow queries to
be true, even though the prechecker said otherwise, allowed the ITSM to go
through...
8.1.02 going to 9.1.03
On Tue, Oct 10, 2017, 6:27 AM Thomas Miskiewicz
<[email protected]<mailto:[email protected]>> wrote:
**
Satya,
the environment is NOT a copy or clone of another environment. The servgrp...
Tables don’t exist.
I did a clean start with deleting logs etc few times already. Nothing new. Same
stuff in the logs.
I would appreciate if you could review my armonitor.conf and
ARSystemInstalledConfiguration.xml Maybe you see something’s I don’t.
Apart from that I have no idea what else I can do. Even worse neither does BMC!
They‘ve been discussing the matter internally since a couple of days but didn’t
approach us with a single suggestion.
When this happens (and this is how BMC support in India works 99.9% of time),
they usually wait two weeks and send me a question whether the problem still
persists. How ridiculous is that!
Thomas
On 6. Oct 2017, at 20:05, Satya Gandhi
<[email protected]<mailto:[email protected]>> wrote:
**
Hi Thomas,
A dbVersion value of 54 indicates that the database has been upgraded
I also see that there is alternating information about the ARServer being
secondary serher AT least when it is validating FTS ports
If the database was a copy from another environment which was part of a server
group, then perform the flowing steps
Remove the old arerror log and armonitor log files and restart the arsystem
services. If the reserves starts successfully the perform the following steps
Delete records from table servgrp_board and servgrp_resources
Change db version value to what is currently on prevdbversion column and
remove the value from prevdbversion
On the AR config file, set multiple ar servers to false and server group member
to false. Also ensure that there are no references for secondary server for any
plugins or any features. Comment those lines out, if you find any
Remove any files from the temp directory to have clean set of install log files
Ensure that the arsystem server licenses are valid and you are able to create
new records in any for that already have more than 2000 entries
Also set the min and max values for private queues, fast and list threads to
the current and recommended maximum value for your infrastructure
Now run the installer and let us know how it goes
Thanks
Satya Gandhi
On 6 Oct 2017 6:29 pm, "Thomas Miskiewicz"
<[email protected]<mailto:[email protected]>> wrote:
**
Hi,
the dbVersion is 54.
Server is not part of the server group, still the stupid installer says: server
is NOT server group member, server IS server group member, server is NOT server
group member … why?
(Oct 06 2017 02:29:12.122 PM
+0200),CONFIG,com.bmc.install.task.InstallationPropertiesHelper,
LOG EVENT {Description=[SET PROPERTY BMC_AR_SERVER_GROUP_MEMBER],Detail=[F]}
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Num-Preload-Schema-Segments, , 25, searchKEYandREPLACEvalue]],
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Num-Preload-Threads, , 10, searchKEYandREPLACEvalue]],
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Application-Enable, F, T, setCFGentry]],
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Large-Result-Logging-Threshold, , 1000000, setCFGentry]],
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Max-Log-History, , 8, setCFGentry]],
[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Max-Log-File-Size, , 134217728, setCFGentry]]]], [level=Database Only Server
Group
Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Server-Group-Member, , F, searchKEYandREPLACEvalue]]]], [level=AR Server Group
Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Server-Group-Member, , T, searchKEYandREPLACEvalue]]]], [level=Configure
JMX,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Jmx-port, , 61500, searchKEYandREPLACEvalue]]]], [level=Configure
JMS,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Default-messaging-port, , 61617, searchKEYandREPLACEvalue]]]], [level=AR
Server Atrium SSO Configurati
(Oct 06 2017 02:47:07.676 PM
+0200),INFO,com.bmc.smbu.install.common.rule.engine.StageGroup,
LOG EVENT {Description=[Skipping execution of stage],Detail=[[level=Database
Only Server Group
Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Server-Group-Member, , F, searchKEYandREPLACEvalue]]]]]}
(Oct 06 2017 02:47:07.676 PM
+0200),INFO,com.bmc.smbu.install.common.rule.engine.Stage,
LOG EVENT {Description=[Executing stage [level=AR Server Group
Configuration,commands=[[className=com.bmc.install.product.arsuitekit.platforms.arsystemservers.task.ARServerUpdateConfigurationEntry,arguments=[/ESP/usr/ar/ARSE1/conf/ar.conf,
Server-Group-Member, , T, searchKEYandREPLACEvalue]]]]]}
(Oct 06 2017 02:47:07.676 PM
+0200),INFO,com.bmc.smbu.install.common.rule.engine.extension.ExtensionCommand,
LOG EVENT {Description=[Executing extension Command]}
etc.
arsystem_install_log.txt attached
ARSystemInstalledConfiguration.xml didn’t exist but it was created after the
first failed installation. In there is say, server is NOT server group member.
So does ar.conf
I’m on Solaris in this case.
arerror.log has a few hundreds entries complaining that he couldn’t import
forms:
Fr Okt 06 15:26:28.836 2017 Import failed for AR System Administration: Add Or
Remove Licenses:
Fr Okt 06 15:26:28.953 2017 Import failed for AR System Administration: Display
Form To Collect User Decisions:
Fr Okt 06 15:26:29.225 2017 Import failed for AR System License: Save Produse
Attachment:
Fr Okt 06 15:26:29.530 2017 Import failed for AR System Licenses Console:
Fr Okt 06 15:26:29.688 2017 Import failed for AR System: Generate License Usage
Report:
Fr Okt 06 15:26:29.986 2017 Import failed for AR System Licenses Audit:
Fr Okt 06 15:26:30.229 2017 Error updating audit shadow form AR System Licenses
Audit
Thomas
_ARSlist: "Where the Answers Are" and have been for 20 years_
**
On 6. Oct 2017, at 19:13, Satya Gandhi
<[email protected]<mailto:[email protected]>> wrote:
**
Hi,
Check the record in control table from a db utility
Select * from control
The value in dbVersion column should be 54 and if that's not the case, the
database has not been upgraded to 9.1.03.
If the AR Server binaries has been upgraded but not the database, t
It should given an error Bout column useShaH (or something similar). This
column is created only when the database has also been upgraded
Was the server part of server group? If yes, can you please check if the
server was considered a secondary server and hence updates only the binaries
and not the database. This can be found on the systeminstalledconfiguration.xml
file
If you are running the server on Windows, check the productregistry.xml to find
the install and upgrade entries for product BMC Remedy action request system
server
I cannot open the log files as I am on my mobile right now. Are there any
glaring errors on arsystem install logs?
Regards
Satya Gandhi
On 6 Oct 2017 18:06, "Grooms, Frederick W"
<[email protected]<mailto:[email protected]>> wrote:
Is there anything in the arDebug.log or arError.log?
-----Original Message-----
From: Action Request System discussion list(ARSList)
[mailto:[email protected]<mailto:[email protected]>] On Behalf Of Thomas
Miskiewicz
Sent: Friday, October 06, 2017 11:24 AM
To: [email protected]<mailto:[email protected]>
Subject: Second try --> Upgrade 8.1.00 to 9.1.03 Failed
Hello Listers,
at the moment I do ARS upgrades from 7.6.4, 8.1.00 and higher to 9.1.03 for
various customers. OS: Linux and Solaris.
Not a single system that just worked! I'm really not sure how to move to
production under this circumstances...
What's even worse: the BMC support in India has zero clue and just keeping us
busy with their ideas and requests. In all the month not a single hint with
substance! BTW, talked to some partners and customers which I do not work for -
same issues! But BMC support always pretends they never heard of any issues.
The logs of my recent
installation:https://s3.eu-central-1.amazonaws.com/miskiewicz/arslist/INC112333.tar.gz
Maybe one of you has an idea what is going on.
Basically the job went like this:
1. Clean and working 8.1.00 installation.
2. Installer 9.1.03 runs.
3. Installer 9.1.03 cannot start the server it installed and gives up.
4. Result: A messed up ARS 9.1.03 installation that doesn't start, but is
trying to do a gazzilion of inserts into various tables and is violating the
unique index at the same time.
I restored and started from scratch. Same result. Not sure what to do. The only
comfort is BMC doesn't know either. I feel less stupid.
Thomas
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at
www.arslist.org<http://www.arslist.org/>
"Where the Answers Are, and have been for 20 years"
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
DISCLAIMER: The information contained in this e-mail and its attachments
contain confidential information belonging to the sender, which is legally
privileged. The information is intended only for the use of the recipient(s)
named above. If you are not the intended recipient, you are notified that any
disclosure, copying, distribution or action in reliance upon the contents of
the information transmitted is strictly prohibited. If you have received this
information in error, please delete it immediately. _ARSlist: "Where the
Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Virus-free.
www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_ARSlist: "Where the Answers Are" and have been for 20 years_
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
"Where the Answers Are, and have been for 20 years"
