Re: [Freeipa-users] pki-tomcat failure
On 11/01/2017 13:55, Petr Vobornik wrote: > On 01/10/2017 09:31 PM, Bob Hinton wrote: >> Hi, >> >> The pki-tomcatd services on our IPA servers seem to have stopped working. >> >> This seems to be related to the expiry of several certificates - >> >> [root@ipa001 ~]# getcert list | more >> Number of certificates and requests being tracked: 8. >> Request ID '20161230150048': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=LOCAL.COM >> subject: CN=CA Audit,O=LOCAL.COM >> expires: 2017-01-09 08:21:45 UTC >> key usage: digitalSignature,nonRepudiation >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20161230150049': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=LOCAL.COM >> subject: CN=OCSP Subsystem,O=LOCAL.COM >> expires: 2017-01-09 08:21:45 UTC >> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign >> eku: id-kp-OCSPSigning >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> >> These were originally in CA_WORKING state, but I moved the clock back >> and restarted certmonger to try to renew them. > > Certs above have: >expires: 2017-01-09 08:21:45 UTC > > But log has 10/Jan so the log is from the time when certs are expired. > > Move time back when all certs reported by `getcert list` are valid. > Restart IPA. Resubmit all certs which are about to expire. Move time back. > Hi Petr, I had already tried moving the clock back, but unfortunately tomcat-pki still wouldn't start. I had to temporarily configure it to connect to LDAP using BasicAuth, as suggested by Adam Tkac. With this done and the time moved back tomcat-pki started OK and restarting certmonger made it renew all the certs. Presumably something was already broken and so certmonger didn't renew them in the first place. Thanks Bob >> >> /var/log/pki/pki-tomcat/ca/debug contains >> >> [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection: >> errorIfDown true >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: Setting desired cert nickname to: >> subsystemCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set >> client auth cert nickname subsystemCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificatSelectionCB: Entering! >> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: >> caSigningCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: >> Server-Cert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: returning: null >> [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened >> Could not connect to LDAP server host ipa001.mgmt.local.com port 636 >> Error netscape.ldap.LDAPException: Authentication failed (48) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) >> at >> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169) >> at >> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075) >> at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571) >> at com.netscape.certsrv.apps.CMS.init(CMS.java:187) >> at com.netscape.certsrv.apps.CMS.start(CMS.java:1616) >> at >> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) >> at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >>
Re: [Freeipa-users] pki-tomcat failure
On 01/10/2017 09:31 PM, Bob Hinton wrote: > Hi, > > The pki-tomcatd services on our IPA servers seem to have stopped working. > > This seems to be related to the expiry of several certificates - > > [root@ipa001 ~]# getcert list | more > Number of certificates and requests being tracked: 8. > Request ID '20161230150048': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=LOCAL.COM > subject: CN=CA Audit,O=LOCAL.COM > expires: 2017-01-09 08:21:45 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20161230150049': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=LOCAL.COM > subject: CN=OCSP Subsystem,O=LOCAL.COM > expires: 2017-01-09 08:21:45 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > > These were originally in CA_WORKING state, but I moved the clock back > and restarted certmonger to try to renew them. Certs above have: expires: 2017-01-09 08:21:45 UTC But log has 10/Jan so the log is from the time when certs are expired. Move time back when all certs reported by `getcert list` are valid. Restart IPA. Resubmit all certs which are about to expire. Move time back. > > > /var/log/pki/pki-tomcat/ca/debug contains > > [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection: > errorIfDown true > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificateSelectionCB: Setting desired cert nickname to: > subsystemCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set > client auth cert nickname subsystemCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificatSelectionCB: Entering! > [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: > caSigningCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: > Server-Cert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificateSelectionCB: returning: null > [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened > Could not connect to LDAP server host ipa001.mgmt.local.com port 636 > Error netscape.ldap.LDAPException: Authentication failed (48) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > at > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169) > at > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075) > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571) > at com.netscape.certsrv.apps.CMS.init(CMS.java:187) > at com.netscape.certsrv.apps.CMS.start(CMS.java:1616) > at > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at >
[Freeipa-users] pki-tomcat failure
Hi, The pki-tomcatd services on our IPA servers seem to have stopped working. This seems to be related to the expiry of several certificates - [root@ipa001 ~]# getcert list | more Number of certificates and requests being tracked: 8. Request ID '20161230150048': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=LOCAL.COM subject: CN=CA Audit,O=LOCAL.COM expires: 2017-01-09 08:21:45 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20161230150049': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=LOCAL.COM subject: CN=OCSP Subsystem,O=LOCAL.COM expires: 2017-01-09 08:21:45 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes These were originally in CA_WORKING state, but I moved the clock back and restarted certmonger to try to renew them. /var/log/pki/pki-tomcat/ca/debug contains [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection: errorIfDown true [10/Jan/2017:18:35:37][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [10/Jan/2017:18:35:37][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: caSigningCert cert-pki-ca [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: Server-Cert cert-pki-ca [10/Jan/2017:18:35:37][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: null [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host ipa001.mgmt.local.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571) at com.netscape.certsrv.apps.CMS.init(CMS.java:187) at com.netscape.certsrv.apps.CMS.start(CMS.java:1616) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
