On 01/10/2017 09:31 PM, Bob Hinton wrote: > Hi, > > The pki-tomcatd services on our IPA servers seem to have stopped working. > > This seems to be related to the expiry of several certificates - > > [root@ipa001 ~]# getcert list | more > Number of certificates and requests being tracked: 8. > Request ID '20161230150048': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=LOCAL.COM > subject: CN=CA Audit,O=LOCAL.COM > expires: 2017-01-09 08:21:45 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20161230150049': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=LOCAL.COM > subject: CN=OCSP Subsystem,O=LOCAL.COM > expires: 2017-01-09 08:21:45 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > track: yes > auto-renew: yes > > These were originally in CA_WORKING state, but I moved the clock back > and restarted certmonger to try to renew them.
Certs above have: expires: 2017-01-09 08:21:45 UTC But log has 10/Jan so the log is from the time when certs are expired. Move time back when all certs reported by `getcert list` are valid. Restart IPA. Resubmit all certs which are about to expire. Move time back. > > > /var/log/pki/pki-tomcat/ca/debug contains > > [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection: > errorIfDown true > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificateSelectionCB: Setting desired cert nickname to: > subsystemCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set > client auth cert nickname subsystemCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificatSelectionCB: Entering! > [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: > caSigningCert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: > Server-Cert cert-pki-ca > [10/Jan/2017:18:35:37][localhost-startStop-1]: > SSLClientCertificateSelectionCB: returning: null > [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened > Could not connect to LDAP server host ipa001.mgmt.local.com port 636 > Error netscape.ldap.LDAPException: Authentication failed (48) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) > at > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) > at > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169) > at > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075) > at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571) > at com.netscape.certsrv.apps.CMS.init(CMS.java:187) > at com.netscape.certsrv.apps.CMS.start(CMS.java:1616) > at > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) > at javax.servlet.GenericServlet.init(GenericServlet.java:158) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) > at > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) > at > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) > at > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) > at > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) > at > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) > at > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > at > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) > at > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) > at > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > at > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) > at > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Internal Database Error encountered: Could not connect to LDAP server > host ipa001.mgmt.local.com port 636 Error netscape.ldap.LDAPException: > Authentication failed (48) > at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) > > The only connection attempt I can find relating to err=48 in the slapd > access log is - > > > [10/Jan/2017:18:21:08.884446519 +0000] conn=59668 fd=83 slot=83 SSL > connection from 10.220.6.250 to 10.220.6.250 > [10/Jan/2017:18:21:08.898844561 +0000] conn=59668 TLS1.2 256-bit AES > [10/Jan/2017:18:21:08.917314723 +0000] conn=59668 op=0 BIND dn="" > method=sasl version=3 mech=EXTERNAL > [10/Jan/2017:18:21:08.919725280 +0000] conn=59668 op=0 RESULT err=48 > tag=97 nentries=0 etime=0 > [10/Jan/2017:18:21:09.590236408 +0000] conn=59637 op=88 EXT > oid="2.16.840.1.113730.3.5.12" name="replication-multimaster-extop" > > We recent upgraded ipa from 4.2 to 4.4 and I wonder if that broke something. > > ipa --version > VERSION: 4.4.0, API_VERSION: 2.213 > > The /etc/ca.crt cert was originally created on an ipa 3.3 server that no > longer exists, I don't know if that's relevant. > > Anyway, I'm stumped on how to fix this so could anyone please help. > > Many thanks > > Bob > -- Petr Vobornik -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
