McCluskey, Chris via FreeIPA-users wrote: > Hello, > > > > I’m hoping somebody here can help with an issue I’m having with upgrades > and ipaca. I have (3) CentOS 7.1.1503 based systems that I’m trying to > upgrade from 4.2.0-15.0.1.el7 to ipa-server-4.5.4-10.el7. I’m able to > upgrade the “second master” (dirsrv, DNS, ipaca backup) and “third > master” (dirsrv, DNS, apace backup) without an issue (replication is > good after 3-4 hours). But when I try to upgrade the “first master” > (dirsrv, DNS, ipaca primary) the upgrade process completes successfully > and starts the services, but the pki-tomcat fails to stay running. Odd > thing is that it does run for about 4-5 minutes (I can see certificate > data, and can list certificates from the CLI), but after about 5 minutes > the whole IPA system stops (per the systemctl). I can run the IPA > services on the “first master” (ipactl start --ignore-service-failures), > but eventually the replication for ipaca fails — I suppose this is > expected since pki-tomcat isn’t running (LDAP connection error from the > “first master” to the “second/third masters”). > > > > Funny thing is I’m not able to see anything in the logs that point to > anything that shows as a fault. In referencing > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/, > I wasn’t able to isolate anything that seems like it was related to the > issue of pki-tomcatd terminating. > > > > I eventually reverted all hosts back to a “safe snapshot” and reverted > to keep the production systems active and in sync. I’m hoping the wise > people here might be able to ID something amiss in the currently running > systems before I make another attempt to get the systems upgraded again, > or perhaps suggest superset of logs and data-points I would need to > gather if it was to fail again. > > > > Per > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ > > — > > > > /var/log/ipaupgrade.log: Completed successfully > > > > /var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings > marked as “error”. > > > > Also checked catalina.out, and didn’t see anything amiss. There was a > trace for a missing module (can’t remember the name right now and it > isn’t in my notes), but the services and webapps started without it. > > > > Certificate cert-pki-ca: > > sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert > cert-pki-ca' | grep 'Not' > > Not Before: Sun Aug 20 22:02:05 2017 > > Not After : Sat Aug 10 22:02:05 2019 > > > > sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n > Server-Cert |grep "Not " > > Not Before: Thu Aug 31 22:02:18 2017 > > Not After : Sun Sep 01 22:02:18 2019 > > > > sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not " > > Not Before: Thu Aug 31 22:02:08 2017 > > Not After : Sun Sep 01 22:02:08 2019 > > > > Was able to read cert using the password in > /var/lib/pki/pki-tomcat/conf/password.conf. > > > > In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be > valid and matches what is in the NSSDB. > > > > There are (8) certificates being monitored, and none have expired — > > > > sudo getcert list > > Number of certificates and requests being tracked: 8. > > Request ID '20150928161427': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed > 2>-COM',nickname='Server-Cert',token='NSS Certificate > DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed 2>-COM/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed > 2>-COM',nickname='Server-Cert',token='NSS Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=starfleet.<removed>.<removed > 2>.com,O=<removed>.<removed 2>.COM > > expires: 2019-09-01 22:02:18 UTC > > principal name: ldap/starfleet.<removed>.<removed > 2>.com@<removed>.<removed 2>.COM > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: > /usr/lib64/ipa/certmonger/restart_dirsrv <removed>-<removed 2>-COM > > track: yes > > auto-renew: yes > > Request ID '20150928161756': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=starfleet.<removed>.<removed > 2>.com,O=<removed>.<removed 2>.COM > > expires: 2019-09-01 22:02:08 UTC > > principal name: HTTP/starfleet.<removed>.<removed > 2>.com@<removed>.<removed 2>.COM > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > > track: yes > > auto-renew: yes > > Request ID '20160725201511': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=CA Audit,O=<removed>.<removed 2>.COM > > expires: 2019-08-10 22:04:31 UTC > > key usage: digitalSignature,nonRepudiation > > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20160725201512': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS > Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS > Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=OCSP Subsystem,O=<removed>.<removed 2>.COM > > expires: 2019-08-10 22:02:50 UTC > > eku: id-kp-OCSPSigning > > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20160725201513': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=CA Subsystem,O=<removed>.<removed 2>.COM > > expires: 2019-08-10 22:02:11 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20160725201514': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=Certificate Authority,O=<removed>.<removed > 2>.COM > > expires: 2035-09-28 16:13:20 UTC > > key usage: > digitalSignature,nonRepudiation,keyCertSign,cRLSign > > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20160725201515': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS > Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=IPA RA,O=<removed>.<removed 2>.COM > > expires: 2019-08-10 22:04:11 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > /usr/lib64/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > Request ID '20160725201516': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-renew-agent > > issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM > > subject: CN=starfleet.<removed>.<removed > 2>.com,O=<removed>.<removed 2>.COM > > expires: 2019-08-10 22:02:05 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > > post-save command: > /usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" > > track: yes > > auto-renew: yes > > > > Replication notes – > > > > ipa server-role-find > > ----------------------- > > 18 server roles matched > > ----------------------- > > Server name: federation.<removed>.<removed 2>.com > > Role name: CA server > > Role status: enabled > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: CA server > > Role status: enabled > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: CA server > > Role status: enabled > > > > Server name: federation.<removed>.<removed 2>.com > > Role name: DNS server > > Role status: enabled > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: DNS server > > Role status: enabled > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: DNS server > > Role status: enabled > > > > Server name: federation.<removed>.<removed 2>.com > > Role name: NTP server > > Role status: enabled > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: NTP server > > Role status: enabled > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: NTP server > > Role status: absent > > > > Server name: federation.<removed>.<removed 2>.com > > Role name: AD trust agent > > Role status: absent > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: AD trust agent > > Role status: absent > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: AD trust agent > > Role status: absent > > > > Server name: federation.<removed>.<removed 2>.com > > Role name: KRA server > > Role status: absent > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: KRA server > > Role status: absent > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: KRA server > > Role status: absent > > > > Server name: federation.<removed>.<removed 2>.com > > Role name: AD trust controller > > Role status: absent > > > > Server name: romulus.<removed>.<removed 2>.com > > Role name: AD trust controller > > Role status: absent > > > > Server name: starfleet.<removed>.<removed 2>.com > > Role name: AD trust controller > > Role status: absent > > > > > > 1) Are there other tests, reports, data that I can perform/provide with > the systems in the “pre-upgrade” state that can help prove out the > “pre-upgrade” state of the systems? I did run > https://pypi.python.org/pypi/checkipaconsistency and cleaned up some > stale RUVs, but they weren’t tried to an active replication agreement > (previous “add” failures for the secondary and tertiary server). > > 2) Are there other points not discussed in > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ > that I should be looking for in the event the service fails to start again? > > 3) Is there a superset of the list of files I should look for when > pki-tomcat fails to start? > > 3a) Is searching for stack traces and “error” (case insensitive) in the > logs sufficient search patterns or are there other searches and data > that need to be looked at?
I think the dogtag debug and selftest logs would be what you need to examine. rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/GPQRWRAVLUO2BFGO33A3OZ3ORVRPBYE5/
