Hello,
I’m hoping somebody here can help with an issue I’m having with upgrades and ipaca. I have (3) CentOS 7.1.1503 based systems that I’m trying to upgrade from 4.2.0-15.0.1.el7 to ipa-server-4.5.4-10.el7. I’m able to upgrade the “second master” (dirsrv, DNS, ipaca backup) and “third master” (dirsrv, DNS, apace backup) without an issue (replication is good after 3-4 hours). But when I try to upgrade the “first master” (dirsrv, DNS, ipaca primary) the upgrade process completes successfully and starts the services, but the pki-tomcat fails to stay running. Odd thing is that it does run for about 4-5 minutes (I can see certificate data, and can list certificates from the CLI), but after about 5 minutes the whole IPA system stops (per the systemctl). I can run the IPA services on the “first master” (ipactl start --ignore-service-failures), but eventually the replication for ipaca fails — I suppose this is expected since pki-tomcat isn’t running (LDAP connection error from the “first master” to the “second/third masters”). Funny thing is I’m not able to see anything in the logs that point to anything that shows as a fault. In referencing https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/, I wasn’t able to isolate anything that seems like it was related to the issue of pki-tomcatd terminating. I eventually reverted all hosts back to a “safe snapshot” and reverted to keep the production systems active and in sync. I’m hoping the wise people here might be able to ID something amiss in the currently running systems before I make another attempt to get the systems upgraded again, or perhaps suggest superset of logs and data-points I would need to gather if it was to fail again. Per https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ — /var/log/ipaupgrade.log: Completed successfully /var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings marked as “error”. Also checked catalina.out, and didn’t see anything amiss. There was a trace for a missing module (can’t remember the name right now and it isn’t in my notes), but the services and webapps started without it. Certificate cert-pki-ca: sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep 'Not' Not Before: Sun Aug 20 22:02:05 2017 Not After : Sat Aug 10 22:02:05 2019 sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n Server-Cert |grep "Not " Not Before: Thu Aug 31 22:02:18 2017 Not After : Sun Sep 01 22:02:18 2019 sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not " Not Before: Thu Aug 31 22:02:08 2017 Not After : Sun Sep 01 22:02:08 2019 Was able to read cert using the password in /var/lib/pki/pki-tomcat/conf/password.conf. In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be valid and matches what is in the NSSDB. There are (8) certificates being monitored, and none have expired — sudo getcert list Number of certificates and requests being tracked: 8. Request ID '20150928161427': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed 2>-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM expires: 2019-09-01 22:02:18 UTC principal name: ldap/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <removed>-<removed 2>-COM track: yes auto-renew: yes Request ID '20150928161756': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM expires: 2019-09-01 22:02:08 UTC principal name: HTTP/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20160725201511': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=CA Audit,O=<removed>.<removed 2>.COM expires: 2019-08-10 22:04:31 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160725201512': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=OCSP Subsystem,O=<removed>.<removed 2>.COM expires: 2019-08-10 22:02:50 UTC eku: id-kp-OCSPSigning pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160725201513': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=CA Subsystem,O=<removed>.<removed 2>.COM expires: 2019-08-10 22:02:11 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160725201514': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=Certificate Authority,O=<removed>.<removed 2>.COM expires: 2035-09-28 16:13:20 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160725201515': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=IPA RA,O=<removed>.<removed 2>.COM expires: 2019-08-10 22:04:11 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20160725201516': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM expires: 2019-08-10 22:02:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Replication notes – ipa server-role-find ----------------------- 18 server roles matched ----------------------- Server name: federation.<removed>.<removed 2>.com Role name: CA server Role status: enabled Server name: romulus.<removed>.<removed 2>.com Role name: CA server Role status: enabled Server name: starfleet.<removed>.<removed 2>.com Role name: CA server Role status: enabled Server name: federation.<removed>.<removed 2>.com Role name: DNS server Role status: enabled Server name: romulus.<removed>.<removed 2>.com Role name: DNS server Role status: enabled Server name: starfleet.<removed>.<removed 2>.com Role name: DNS server Role status: enabled Server name: federation.<removed>.<removed 2>.com Role name: NTP server Role status: enabled Server name: romulus.<removed>.<removed 2>.com Role name: NTP server Role status: enabled Server name: starfleet.<removed>.<removed 2>.com Role name: NTP server Role status: absent Server name: federation.<removed>.<removed 2>.com Role name: AD trust agent Role status: absent Server name: romulus.<removed>.<removed 2>.com Role name: AD trust agent Role status: absent Server name: starfleet.<removed>.<removed 2>.com Role name: AD trust agent Role status: absent Server name: federation.<removed>.<removed 2>.com Role name: KRA server Role status: absent Server name: romulus.<removed>.<removed 2>.com Role name: KRA server Role status: absent Server name: starfleet.<removed>.<removed 2>.com Role name: KRA server Role status: absent Server name: federation.<removed>.<removed 2>.com Role name: AD trust controller Role status: absent Server name: romulus.<removed>.<removed 2>.com Role name: AD trust controller Role status: absent Server name: starfleet.<removed>.<removed 2>.com Role name: AD trust controller Role status: absent 1) Are there other tests, reports, data that I can perform/provide with the systems in the “pre-upgrade” state that can help prove out the “pre-upgrade” state of the systems? I did run https://pypi.python.org/pypi/checkipaconsistency and cleaned up some stale RUVs, but they weren’t tried to an active replication agreement (previous “add” failures for the secondary and tertiary server). 2) Are there other points not discussed in https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ that I should be looking for in the event the service fails to start again? 3) Is there a superset of the list of files I should look for when pki-tomcat fails to start? 3a) Is searching for stack traces and “error” (case insensitive) in the logs sufficient search patterns or are there other searches and data that need to be looked at? Thank you in advance for the assistance, Chris
_______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/Q6755GRAW6QW65PYJUAMLXD2D7YFEGMR/
