Hi Adam, With the change to ldap instead of ldaps on the CA master that you suggested I was able to move the system clock to before the certificate expiry time then do
ipactl start --ignore-service-failures systemctl start pki-tomcat@pki-tomcat.service then start the pki ca service manually as stated in "systemctl status pki-tomcat@pki-tomcat.service -l" service certmonger restart I then ran "getcert list | more" and waited until all the certificates had updated After that "ipactl stop; ipactl start" completed without errors and I could reenable ntpd and vmware tools timesync. Finally ipa-certupdate seems to have been needed to propagate the new certs to the other replicas. Many thanks Bob On 10/01/2017 20:47, Adam Tkac wrote: > Hello, > > we hit similar issue (although due to different conditions - we rotated > root CA cert and then newly issued certificates were wrongly signed), we > were also unable to start tomcat. If I remember correctly, we switched dogtag > to use simple binds instead of TLS to connect to LDAP this way. > > 1. systemctl stop pki-tomcatd@pki-tomcat.service > 2. backup /etc/pki/pki-tomcat/ca/CS.cfg and /etc/pki/pki-tomcat/password.conf > 3. add directory manager password into password.conf file, it should be line > like > > internaldb=my_directory_manager_password > > 4. tune CS.cfg a little, take this diff as an example > > +internaldb.ldapauth.authtype=BasicAuth > +internaldb.ldapauth.bindDN=cn=Directory Manager > +internaldb.ldapauth.bindPWPrompt=internaldb > -internaldb.ldapauth.authtype=SslClientAuth > -internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca > internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca > internaldb.ldapconn.cloneReplicationPort=389 > internaldb.ldapconn.masterReplicationPort=7389 > +internaldb.ldapconn.port=389 > -internaldb.ldapconn.port=636 > internaldb.ldapconn.replicationSecurity=TLS > +internaldb.ldapconn.secureConn=false > -internaldb.ldapconn.secureConn=true > > 5. systemctl start pki-tomcatd@pki-tomcat.service > > Now tomcat should run correctly and you should be able to resubmit expired > certs and you can start to experiment with switch dogtag back to TLS auth. > Hope this helps you. > > Regards, Adam > > On Tue, Jan 10, 2017 at 07:27:04PM +0000, Bob Hinton wrote: >> Hi, >> >> The pki-tomcatd services on our IPA servers seem to have stopped working. >> >> This seems to be related to the expiry of several certificates - >> >> [root@ipa001 ~]# getcert list | more >> Number of certificates and requests being tracked: 8. >> Request ID '20161230150048': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=LOCAL.COM >> subject: CN=CA Audit,O=LOCAL.COM >> expires: 2017-01-09 08:21:45 UTC >> key usage: digitalSignature,nonRepudiation >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "auditSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> Request ID '20161230150049': >> status: MONITORING >> stuck: no >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB',pin set >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert >> cert-pki-ca',token='NSS Certificate DB' >> CA: dogtag-ipa-ca-renew-agent >> issuer: CN=Certificate Authority,O=LOCAL.COM >> subject: CN=OCSP Subsystem,O=LOCAL.COM >> expires: 2017-01-09 08:21:45 UTC >> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign >> eku: id-kp-OCSPSigning >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> track: yes >> auto-renew: yes >> >> These were originally in CA_WORKING state, but I moved the clock back >> and restarted certmonger to try to renew them. >> >> >> /var/log/pki/pki-tomcat/ca/debug contains >> >> [10/Jan/2017:18:35:37][localhost-startStop-1]: makeConnection: >> errorIfDown true >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: Setting desired cert nickname to: >> subsystemCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: LdapJssSSLSocket: set >> client auth cert nickname subsystemCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificatSelectionCB: Entering! >> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: >> caSigningCert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: Candidate cert: >> Server-Cert cert-pki-ca >> [10/Jan/2017:18:35:37][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: returning: null >> [10/Jan/2017:18:35:37][localhost-startStop-1]: SSL handshake happened >> Could not connect to LDAP server host ipa001.mgmt.local.com port 636 >> Error netscape.ldap.LDAPException: Authentication failed (48) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) >> at >> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) >> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) >> at >> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169) >> at >> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075) >> at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:571) >> at com.netscape.certsrv.apps.CMS.init(CMS.java:187) >> at com.netscape.certsrv.apps.CMS.start(CMS.java:1616) >> at >> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) >> at javax.servlet.GenericServlet.init(GenericServlet.java:158) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) >> at >> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) >> at >> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) >> at >> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) >> at >> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) >> at >> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) >> at >> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) >> at >> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) >> at >> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) >> at >> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) >> at >> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> at >> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) >> at >> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) >> at >> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) >> at >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) >> at >> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) >> at >> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> Internal Database Error encountered: Could not connect to LDAP server >> host ipa001.mgmt.local.com port 636 Error netscape.ldap.LDAPException: >> Authentication failed (48) >> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) >> >> The only connection attempt I can find relating to err=48 in the slapd >> access log is - >> >> >> [10/Jan/2017:18:21:08.884446519 +0000] conn=59668 fd=83 slot=83 SSL >> connection from 10.220.6.250 to 10.220.6.250 >> [10/Jan/2017:18:21:08.898844561 +0000] conn=59668 TLS1.2 256-bit AES >> [10/Jan/2017:18:21:08.917314723 +0000] conn=59668 op=0 BIND dn="" >> method=sasl version=3 mech=EXTERNAL >> [10/Jan/2017:18:21:08.919725280 +0000] conn=59668 op=0 RESULT err=48 >> tag=97 nentries=0 etime=0 >> [10/Jan/2017:18:21:09.590236408 +0000] conn=59637 op=88 EXT >> oid="2.16.840.1.113730.3.5.12" name="replication-multimaster-extop" >> >> We recent upgraded ipa from 4.2 to 4.4 and I wonder if that broke something. >> >> ipa --version >> VERSION: 4.4.0, API_VERSION: 2.213 >> >> The /etc/ca.crt cert was originally created on an ipa 3.3 server that no >> longer exists, I don't know if that's relevant. >> >> Anyway, I'm stumped on how to fix this so could anyone please help. >> >> Many thanks >> >> Bob >> >> -- >> Manage your subscription for the Freeipa-users mailing list: >> https://www.redhat.com/mailman/listinfo/freeipa-users >> Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
