TomK via FreeIPA-users wrote: > Hey All, > > I've upgrade one side of my two node cluster. However, the secondary > won't come even though the manual upgrade apparently went well. > > [root@idmipa04 ~]# ipa-server-upgrade > Upgrading IPA:. Estimated time: 1 minute 30 seconds > [1/9]: saving configuration > [2/9]: disabling listeners > [3/9]: enabling DS global lock > [4/9]: disabling Schema Compat > [5/9]: starting directory server > [6/9]: updating schema > [7/9]: upgrading server > [8/9]: stopping directory server > [9/9]: restoring configuration > Done. > Update complete > Upgrading IPA services > Upgrading the configuration of the IPA services > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run > command ipa-server-upgrade manually. > Unexpected error - see /var/log/ipaupgrade.log for details: > CalledProcessError: Command '/bin/systemctl start > dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for > more information > [root@idmipa04 ~]# > > > Additional information: > > > [root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50 > 2020-05-19T04:18:10Z DEBUG Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Saving StateFile to > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Saving StateFile to > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG duration: 0 seconds > 2020-05-19T04:18:10Z DEBUG Done. > 2020-05-19T04:18:10Z INFO Update complete > 2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services > 2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos > 2020-05-19T04:18:10Z DEBUG Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > 2020-05-19T04:18:10Z DEBUG Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > 2020-05-19T04:18:10Z DEBUG Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > 2020-05-19T04:18:10Z DEBUG Starting external process > 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active > dirsrv@MWS-MDS-XYZ.service > 2020-05-19T04:18:10Z DEBUG Process finished, return code=3 > 2020-05-19T04:18:10Z DEBUG stdout=unknown > > 2020-05-19T04:18:10Z DEBUG stderr= > 2020-05-19T04:18:10Z DEBUG Starting external process > 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start > dirsrv@MWS-MDS-XYZ.service > 2020-05-19T04:19:55Z DEBUG Process finished, return code=1 > 2020-05-19T04:19:55Z DEBUG stdout= > 2020-05-19T04:19:55Z DEBUG stderr=Job for dirsrv@MWS-MDS-XYZ.service > failed because a fatal signal was delivered to the control process. See > "systemctl status dirsrv@MWS-MDS-XYZ.service" and "journalctl -xe" for > details. > > 2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > 2020-05-19T04:19:56Z DEBUG File > "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in > execute > return_value = self.run() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", > line 54, in run > server.upgrade() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", > line 2166, in upgrade > upgrade_configuration() > File > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", > line 1791, in upgrade_configuration > ds.start(ds_serverid) > File > "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line > 656, in start > super(DsInstance, self).start(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", > line 464, in start > self.service.start(instance_name, capture_output=capture_output, > wait=wait) > File > "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line > 136, in start > instance_name, capture_output=capture_output, wait=wait) > File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", > line 303, in start > skip_output=not capture_output) > File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line > 563, in run > raise CalledProcessError(p.returncode, arg_string, str(output)) > > 2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed, > exception: CalledProcessError: Command '/bin/systemctl start > dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 > 2020-05-19T04:19:56Z ERROR Unexpected error - see > /var/log/ipaupgrade.log for details: > CalledProcessError: Command '/bin/systemctl start > dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 > 2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See > /var/log/ipaupgrade.log for more information > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# /bin/systemctl start dirsrv@MWS-MDS-XYZ.service > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service > ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. > Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; > vendor preset: disabled) > Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago > Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl > /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) > Main PID: 4664 (ns-slapd) > Status: "slapd started: Ready to process requests" > CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service > └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i > /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid > > May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm) > May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) () > May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds! > May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm) > May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm) > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion. > May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm) > May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm) > Hint: Some lines were ellipsized, use -l to show in full. > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# > [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service -l > ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. > Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; > vendor preset: disabled) > Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago > Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl > /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) > Main PID: 4664 (ns-slapd) > Status: "slapd started: Ready to process requests" > CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service > └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i > /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid > > May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could > not get initial credentials for principal > [ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could > not get initial credentials for principal > [ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin - > warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin - > warning: no entries set up under cn=computers, > cn=compat,dc=mws,dc=mds,dc=xyz > May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin - > Finished plugin initialization. > May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could > not get initial credentials for principal > [ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: > [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could > not get initial credentials for principal > [ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for > requested realm) > May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1 > May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2 > May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3 > [root@idmipa04 ~]# > [root@idmipa04 ~]# > > > --------------------------------------------------------------------- > > Looking at the service, appears libc hit a SEGFAULT. > > > [root@idmipa04 ~]# journalctl -xe > May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: > /usr/bin/killall -0 haproxy exited with status 1 > May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: > /usr/bin/killall -0 haproxy exited with status 1 > May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]: > [19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin > - changelog program - _cl5ConstructRU > May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at > 5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in > libc-2.17.so[7fe3ba242000 > May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: > dirsrv@MWS-MDS-XYZ.service: main process exited, code=killed, > status=11/SEGV > May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389 > Directory Server MWS-MDS-XYZ.. > -- Subject: Unit dirsrv@MWS-MDS-XYZ.service has failed > -- Defined-By: systemd > -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel > -- > -- Unit dirsrv@MWS-MDS-XYZ.service has failed. > -- > -- The result is failed. > > > Wondering what should my next steps be from here?
Install the debuginfo packages and get a stack trace. https://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
