On 5/19/2020 8:21 AM, Rob Crittenden via FreeIPA-users wrote:
TomK via FreeIPA-users wrote:
Hey All,
I've upgrade one side of my two node cluster. However, the secondary
won't come even though the manual upgrade apparently went well.
[root@idmipa04 ~]# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/9]: saving configuration
[2/9]: disabling listeners
[3/9]: enabling DS global lock
[4/9]: disabling Schema Compat
[5/9]: starting directory server
[6/9]: updating schema
[7/9]: upgrading server
[8/9]: stopping directory server
[9/9]: restoring configuration
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start
dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
[root@idmipa04 ~]#
Additional information:
[root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Saving StateFile to
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG duration: 0 seconds
2020-05-19T04:18:10Z DEBUG Done.
2020-05-19T04:18:10Z INFO Update complete
2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services
2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos
2020-05-19T04:18:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2020-05-19T04:18:10Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2020-05-19T04:18:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2020-05-19T04:18:10Z DEBUG Starting external process
2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active
dirsrv@MWS-MDS-XYZ.service
2020-05-19T04:18:10Z DEBUG Process finished, return code=3
2020-05-19T04:18:10Z DEBUG stdout=unknown
2020-05-19T04:18:10Z DEBUG stderr=
2020-05-19T04:18:10Z DEBUG Starting external process
2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start
dirsrv@MWS-MDS-XYZ.service
2020-05-19T04:19:55Z DEBUG Process finished, return code=1
2020-05-19T04:19:55Z DEBUG stdout=
2020-05-19T04:19:55Z DEBUG stderr=Job for dirsrv@MWS-MDS-XYZ.service
failed because a fatal signal was delivered to the control process. See
"systemctl status dirsrv@MWS-MDS-XYZ.service" and "journalctl -xe" for
details.
2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2020-05-19T04:19:56Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
server.upgrade()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2166, in upgrade
upgrade_configuration()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1791, in upgrade_configuration
ds.start(ds_serverid)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line
656, in start
super(DsInstance, self).start(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
line 464, in start
self.service.start(instance_name, capture_output=capture_output,
wait=wait)
File
"/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
136, in start
instance_name, capture_output=capture_output, wait=wait)
File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
line 303, in start
skip_output=not capture_output)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
563, in run
raise CalledProcessError(p.returncode, arg_string, str(output))
2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed,
exception: CalledProcessError: Command '/bin/systemctl start
dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1
2020-05-19T04:19:56Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start
dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1
2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# /bin/systemctl start dirsrv@MWS-MDS-XYZ.service
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service
● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago
Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl
/etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 4664 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
└─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i
/var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) ()
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds!
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion.
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm)
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm)
Hint: Some lines were ellipsized, use -l to show in full.
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]#
[root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service -l
● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled;
vendor preset: disabled)
Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago
Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl
/etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 4664 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service
└─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i
/var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin -
warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin -
warning: no entries set up under cn=computers,
cn=compat,dc=mws,dc=mds,dc=xyz
May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin -
Finished plugin initialization.
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]:
[19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/idmipa04.mws.mds....@mws.mds.xyz] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2
May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3
[root@idmipa04 ~]#
[root@idmipa04 ~]#
---------------------------------------------------------------------
Looking at the service, appears libc hit a SEGFAULT.
[root@idmipa04 ~]# journalctl -xe
May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]:
/usr/bin/killall -0 haproxy exited with status 1
May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]:
/usr/bin/killall -0 haproxy exited with status 1
May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]:
[19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin
- changelog program - _cl5ConstructRU
May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at
5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in
libc-2.17.so[7fe3ba242000
May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]:
dirsrv@MWS-MDS-XYZ.service: main process exited, code=killed,
status=11/SEGV
May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389
Directory Server MWS-MDS-XYZ..
-- Subject: Unit dirsrv@MWS-MDS-XYZ.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit dirsrv@MWS-MDS-XYZ.service has failed.
--
-- The result is failed.
Wondering what should my next steps be from here?
Install the debuginfo packages and get a stack trace.
https://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
Couldn't get the debug tools to install on this CentOS 7. Had a closer
look at all the log files and what I saw was memory allocation issues:
[root@idmipa04 slapd-MWS-MDS-XYZ]# [root@idmipa04 slapd-MWS-MDS-XYZ]#
cat errors|tail -n 100^C
[root@idmipa04 slapd-MWS-MDS-XYZ]# pwd
/var/log/dirsrv/slapd-MWS-MDS-XYZ
[root@idmipa04 slapd-MWS-MDS-XYZ]#
[root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 30
[23/May/2020:16:33:18.519974074 -0400] - WARN - NSACLPlugin - acl_parse
- The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.522332851 -0400] - WARN - NSACLPlugin - acl_parse
- The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist
[23/May/2020:16:33:18.759212393 -0400] - WARN - NSACLPlugin - acl_parse
- The ACL target cn=automember rebuild membership,cn=tasks,cn=config
does not exist
[23/May/2020:16:33:18.773571691 -0400] - ERR - cos-plugin -
cos_dn_defs_cb - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=mws,dc=mds,dc=xyz--no CoS Templates found, which
should be added before the CoS Definition.
[23/May/2020:16:33:18.820082920 -0400] - NOTICE - NSMMReplicationPlugin
- changelog program - _cl5ConstructRUV - Rebuilding the replication
changelog RUV, this may take several minutes...
[23/May/2020:16:39:06.851785150 -0400] - ERR - memory allocator - malloc
of 2152941454 bytes failed; OS error 12 (Cannot allocate memory)
The server has probably allocated all available virtual memory. To solve
this problem, make more virtual memory available to your server, or reduce
one or more of the following server configuration settings:
nsslapd-cachesize (Database Settings - Maximum entries in cache)
nsslapd-cachememsize (Database Settings - Memory available for cache)
nsslapd-dbcachesize (LDBM Plug-in Settings - Maximum cache size)
nsslapd-import-cachesize (LDBM Plug-in Settings - Import cache size).
Can't recover; calling exit(1).
[root@idmipa04 slapd-MWS-MDS-XYZ]# free -g
total used free shared buff/cache
available
Mem: 1 0 1 0 0
1
Swap: 3 0 3
[root@idmipa04 slapd-MWS-MDS-XYZ]# free -m
total used free shared buff/cache
available
Mem: 1838 120 1520 2 196
1574
Swap: 3967 175 3792
[root@idmipa04 slapd-MWS-MDS-XYZ]# cat /proc/meminfo |grep -Ei
"inactive|active"
Active: 11216 kB
Inactive: 174824 kB
Active(anon): 2372 kB
Inactive(anon): 10396 kB
Active(file): 8844 kB
Inactive(file): 164428 kB
[root@idmipa04 slapd-MWS-MDS-XYZ]#
Posting back in case someone else has the same issue.
rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
--
Thx,
TK.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
