Greetings all,
For the record,this is a CentOS 7.2 box with all current patches.
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)
The situation is that pki-tomcatd on the lone CA server in our IPA cluster
refuses to start cleanly. The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired
without warning; apparently, certmonger failed to renew them automatically. We
attempted timeshifting and following instructions for what appeared to be
similar issues, but nothing at all has worked.
Today, we attempted removing the certificates in question (of course, the files in
/etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new
certificates. This process worked but pki-tomcatd is still refusing to start. We can
get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and
then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a
tenable long-term solution.
Relevant log entries/information:
/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host ipa01.XXXXXXXXX.net port 636 Error
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host
ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: IO Error
creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host
ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: Authentication
failed (49)
/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException
# getcert list:
Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using
default keytab: Generic error (see e-text).
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXX-NET',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-XXXXXXXXX-NET/pwdfile.txt'
expires: 2017-10-15 02:09:06 UTC
track: yes
auto-renew: yes
Request ID '20151015022949':
status: MONITORING
ca-error: Error setting up ccache for "host" service on client using
default keytab: Generic error (see e-text).
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
expires: 2017-10-15 02:09:10 UTC
track: yes
auto-renew: yes
Request ID '20160127202548':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2034-02-11 19:46:43 UTC
track: yes
auto-renew: yes
Request ID '20160127202549':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
expires: 2017-12-25 04:27:49 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
track: yes
auto-renew: yes
Request ID '20160127202550':
status: MONITORING
ca-error: Server at
"http://ipa01.XXXXXXXXX.net:8080/ca/ee/ca/profileSubmit"; replied: Profile
caServerCert Not Found
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
expires: 2017-10-04 02:28:53 UTC
track: yes
auto-renew: yes
Request ID '20160204165453':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:40:23 UTC
track: yes
auto-renew: yes
Request ID '20160204170246':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 16:59:18 UTC
track: yes
auto-renew: yes
Request ID '20160204170752':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2016-05-04 17:05:29 UTC
track: yes
auto-renew: yes
# certutil -L -d /var/lib/pki/pki-tomcat/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca u,u,Pu
ocspSigningCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
subsystemCert cert-pki-ca u,u,u
Server-Cert cert-pki-ca u,u,u
# certutil -L -d /etc/dirsrv/slapd-XXXXXXXXX-NET/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert
u,u,u
XXXXXXXXX.NET IPA CA CT,C,C
The only thing that making new certs seemed to resolve was removing these
errors from /var/log/pki/pki-tomcat/ca/system :
Cannot authenticate agent with certificate Serial <redacted> Subject DN CN=IPA
RA,O=XXXXXXXXX.NET. Error: User not found
Thus, the root cause(s) appears to be something else entirely that we are
totally unfamilar with..we can provide any other required information to help
with troubleshooting.
