Hey All,
I've upgrade one side of my two node cluster. However, the secondary
won't come even though the manual upgrade apparently went well.
[root@idmipa04 ~]# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/9]: saving configuration
[2/9]: disabling listeners
[3/9]: enabling DS global lock
[4/9]: disabling Schema Compat
[5/9]: starting directory server
[6/9]: updating schema
[7/9]: upgrading server
[8/9]: stopping directory server
[9/9]: restoring configuration
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start
[email protected]' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
[root@idmipa04 ~]#
Additional information:
[root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG duration: 0 seconds
2020-05-19T04:18:10Z DEBUG Done.
2020-05-19T04:18:10Z INFO Update complete
2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services
2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos
2020-05-19T04:18:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2020-05-19T04:18:10Z DEBUG Starting external process
2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active
[email protected]
2020-05-19T04:18:10Z DEBUG Process finished, return code=3
2020-05-19T04:18:10Z DEBUG stdout=unknown
2020-05-19T04:18:10Z DEBUG stderr=
2020-05-19T04:18:10Z DEBUG Starting external process
2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start
[email protected]
2020-05-19T04:19:55Z DEBUG Process finished, return code=1
2020-05-19T04:19:55Z DEBUG stdout=
2020-05-19T04:19:55Z DEBUG stderr=Job for [email protected]
failed because a fatal signal was delivered to the control process. See
"systemctl status [email protected]" and "journalctl -xe" for
details.
2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-19T04:19:56Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
server.upgrade()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2166, in upgrade
upgrade_configuration()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1791, in upgrade_configuration
ds.start(ds_serverid)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line
656, in start
super(DsInstance, self).start(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
line 464, in start
self.service.start(instance_name, capture_output=capture_output,
wait=wait)
File
"/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
136, in start
instance_name, capture_output=capture_output, wait=wait)
File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
line 303, in start
skip_output=not capture_output)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
563, in run
raise CalledProcessError(p.returncode, arg_string, str(output))
2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed,
exception: CalledProcessError: Command '/bin/systemctl start
[email protected]' returned non-zero exit status 1
2020-05-19T04:19:56Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start
[email protected]' returned non-zero exit status 1
2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# /bin/systemctl start [email protected]
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# systemctl status [email protected]
● [email protected] - 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago
Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl
/etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 4664 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/[email protected]
└─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ
-i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) ()
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds!
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion.
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm)
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# systemctl status [email protected] -l
● [email protected] - 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/[email protected]; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago
Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl
/etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 4664 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/[email protected]
└─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ
-i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin -
warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin -
warning: no entries set up under cn=computers,
cn=compat,dc=mws,dc=mds,dc=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin -
Finished plugin initialization.
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3
[root@idmipa04 ~]#
[root@idmipa04 ~]#
---------------------------------------------------------------------
Looking at the service, appears libc hit a SEGFAULT.
[root@idmipa04 ~]# journalctl -xe
May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]:
/usr/bin/killall -0 haproxy exited with status 1
May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]:
/usr/bin/killall -0 haproxy exited with status 1
May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]:
[19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin
- changelog program - _cl5ConstructRU
May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at
5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in
libc-2.17.so[7fe3ba242000
May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]:
[email protected]: main process exited, code=killed, status=11/SEGV
May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389
Directory Server MWS-MDS-XYZ..
-- Subject: Unit [email protected] has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit [email protected] has failed.
--
-- The result is failed.
Wondering what should my next steps be from here?
--
Thx,
TK.
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
