Timothy Geier wrote:
Greetings all,

For the record,this is a CentOS 7.2 box with all current patches. 
(ipa-server-4.2.0-15.el7.centos.3.x86_64, etc.)

The situation is that pki-tomcatd on the lone CA server in our IPA cluster 
refuses to start cleanly.  The issues started earlier this week after the certs
subsystemCert, ocspSigningCert, and auditSigningCert all simultaneously expired 
without warning; apparently, certmonger failed to renew them automatically.  We
attempted timeshifting and following instructions for what appeared to be 
similar issues, but nothing at all has worked.

Today, we attempted removing the certificates in question (of course, the files in 
/etc/pki/pki-tomcat/alias were backed up beforehand) and using certutil to issue new  
certificates.   This process worked but pki-tomcatd is still refusing to start.  We can 
get IPA to run on this server by manually starting pki-tomcatd, running ipactl start, and 
then ctrl-c’ing it when it gets to "Starting pki-tomcatd" but this is not a 
tenable long-term solution.

Relevant log entries/information:

/var/log/pki/pki-tomcat/ca/debug:
Could not connect to LDAP server host ipa01.XXXXXXXXX.net port 636 Error 
netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: IO Error 
creating JSS SSL Socket (-1)
Internal Database Error encountered: Could not connect to LDAP server host 
ipa01.XXXXXXXXX.net port 636 Error netscape.ldap.LDAPException: Authentication 
failed (49)

/var/log/pki/pki-tomcat/localhost.2016-02-04.log:
org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /ca threw load() exception
java.lang.NullPointerException

# getcert list:

Number of certificates and requests being tracked: 8.
Request ID '20151015022737':
        status: MONITORING
        ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXX-NET',nickname='Server-Cert',token='NSS
 Certificate DB',pinfile='/etc/dirsrv/slapd-XXXXXXXXX-NET/pwdfile.txt'
        expires: 2017-10-15 02:09:06 UTC
        track: yes
        auto-renew: yes
Request ID '20151015022949':
        status: MONITORING
        ca-error: Error setting up ccache for "host" service on client using 
default keytab: Generic error (see e-text).
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
        expires: 2017-10-15 02:09:10 UTC
        track: yes
        auto-renew: yes
Request ID '20160127202548':
        status: MONITORING
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
        expires: 2034-02-11 19:46:43 UTC
        track: yes
        auto-renew: yes
Request ID '20160127202549':
        status: MONITORING
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS 
Certificate DB'
        expires: 2017-12-25 04:27:49 UTC
        key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        track: yes
        auto-renew: yes
Request ID '20160127202550':
        status: MONITORING
        ca-error: Server at 
"http://ipa01.XXXXXXXXX.net:8080/ca/ee/ca/profileSubmit"; replied: Profile 
caServerCert Not Found
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
        expires: 2017-10-04 02:28:53 UTC
        track: yes
        auto-renew: yes
Request ID '20160204165453':
        status: MONITORING
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
        expires: 2016-05-04 16:40:23 UTC
        track: yes
        auto-renew: yes
Request ID '20160204170246':
        status: MONITORING
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
        expires: 2016-05-04 16:59:18 UTC
        track: yes
        auto-renew: yes
Request ID '20160204170752':
        status: MONITORING
        stuck: no
        key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
        expires: 2016-05-04 17:05:29 UTC
        track: yes
        auto-renew: yes

# certutil -L -d /var/lib/pki/pki-tomcat/alias/

Certificate Nickname                                         Trust Attributes
                                                              SSL,S/MIME,JAR/XPI
auditSigningCert cert-pki-ca                                 u,u,Pu
ocspSigningCert cert-pki-ca                                  u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
subsystemCert cert-pki-ca                                    u,u,u
Server-Cert cert-pki-ca                                      u,u,u

# certutil -L -d /etc/dirsrv/slapd-XXXXXXXXX-NET/

Certificate Nickname                                         Trust Attributes
                                                              SSL,S/MIME,JAR/XPI
Server-Cert                                                                 
u,u,u
XXXXXXXXX.NET IPA CA                                         CT,C,C



The only thing that making new certs seemed to resolve was removing these 
errors from /var/log/pki/pki-tomcat/ca/system :

Cannot authenticate agent with certificate Serial <redacted> Subject DN CN=IPA 
RA,O=XXXXXXXXX.NET. Error: User not found

Thus, the root cause(s) appears to be something else entirely that we are 
totally unfamilar with..we can provide any other required information to help 
with troubleshooting.

It appears that the CA is not fully starting, perhaps due to these renewal issues, perhaps something else. You'll need to dig into the logs. I'd start with /var/lib/pki/pki-ca/pki-tomcat/logs/debug and selftests.log.

You mentioned privately that you renamed the IPA host. This is probably what broke half of the renewals. The hosts and keytabs and many entries in IPA have the hostname baked in so you can't simply rename the host.

rob

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to