More updates.
I rolled back the system clock and set the date to a few days ago. I was
able to get all the IPA services started again. Magically 9 certificates
appeared and my ips1 server turns out to be the natural ca-replication
master. Things are looking better. ipa2 is a VM so I rolled the failed
updates back using a snapshot and then ran ipa-server-upgrade on ipa1.
It actually completed successfully I'm now version 4.6.4. I can access
the webgui and kinit again. I'm still getting errors and a few
certificates are having issues but I hope this is progress.
Any suggestions on how to proceed?
Here are some scary errors from the /var/log/messages:
Feb 5 11:00:55 ipa1 dogtag-ipa-ca-renew-agent-submit: Forwarding
request to dogtag-ipa-renew-agent
Feb 5 11:00:56 ipa1 dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
Feb 5 11:00:56 ipa1 certmonger: 2019-02-05 11:00:56 [3631] Error 58
connecting to https://ipa1.domain.com:8443/ca/agent/ca/profileReview:
Problem with the local SSL certificate.
And the errors from getcert list:
Request ID '20180131032610':
status: CA_UNREACHABLE
ca-error: Error 58 connecting to
https://ipa1.domain.com.edu:8443/ca/agent/ca/profileReview: Problem
with the local SSL certificate.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=domain.com.EDU
subject: CN=CA Audit,O=domain.com.EDU
expires: 2018-12-31 13:28:03 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180131032614':
status: CA_UNREACHABLE
ca-error: Error 58 connecting to
https://ipa1.domain.com.edu:8443/ca/agent/ca/profileReview: Problem
with the local SSL certificate.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=domain.com.EDU
subject: CN=OCSP Subsystem,O=domain.com.EDU
expires: 2018-12-31 13:26:43 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180131032615':
status: CA_UNREACHABLE
ca-error: Error 58 connecting to
https://ipa1.domain.com.edu:8443/ca/agent/ca/profileReview: Problem
with the local SSL certificate.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=domain.com.EDU
subject: CN=CA Subsystem,O=domain.com.EDU
expires: 2018-12-31 13:26:53 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
I think if I can get these cert issues fixed the errors in the logs will
go away and I'll have a working system again. I can then scrap the
replicas I have and just make new replicas once things are stable again.
Thanks,
-Chris
On 2/4/2019 6:11 PM, Chris Mohler wrote:
Well... That was a mess.
The ipa-server-upgrade didn't go so well. It failed and now my
ca-replication master is broken. Here are the details. Any hope?
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/11]: stopping directory server
[2/11]: saving configuration
[3/11]: disabling listeners
[4/11]: enabling DS global lock
[5/11]: disabling Schema Compat
[6/11]: starting directory server
[7/11]: updating schema
[8/11]: upgrading server
[9/11]: stopping directory server
[10/11]: restoring configuration
[11/11]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
for more information
Here is a wall of errors from my /var/log/ipaupgrade.log
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]
- ERR - slapi_ldap_bind - Error: could not send startTLS request:
error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is
not connected)
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS
failure. Minor code may provide more information (No Kerberos
credentials available (default cache: /tmp/krb5cc_389))
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]
- ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
Manager
masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config]
authentication mechanism [SIMPLE]: error 32 (No such object)
Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:47:40 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:47:50 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:00 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:10 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize
credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP connection.
Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize
credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
any KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP connection.
Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:20 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:30 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
^C
[root@ipa2 log]# less /var/log/ipaupgrade.log
<p><b>note</b> <u>The full stack trace of the root cause is available
in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:13Z DEBUG Waiting for CA to start...
2019-02-04T22:46:14Z DEBUG request POST
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
2019-02-04T22:46:14Z DEBUG request body ''
2019-02-04T22:46:14Z DEBUG response status 500
2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Mon, 04 Feb 2019 22:46:14 GMT
Connection: close
2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:14Z DEBUG Waiting for CA to start...
2019-02-04T22:46:15Z DEBUG request POST
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
2019-02-04T22:46:15Z DEBUG request body ''
2019-02-04T22:46:15Z DEBUG response status 500
2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Mon, 04 Feb 2019 22:46:15 GMT
Connection: close
2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:15Z DEBUG Waiting for CA to start...
2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-02-04T22:46:16Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178,
in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 56, in run
raise admintool.ScriptError(str(e))
2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
2019-02-04T22:46:16Z ERROR CA did not start in 300.0s
2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
Thanks,
-Chris
Rob,
I'll be honest. I think you are suggesting an ldapsearch with this
Check to see which masteris the renewal master. Look in
cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
ipaConfigString=caRenewalMaster
sorry I've not figured out how to successfully ldapsearch :-(
Instead I did this:
ipa config-show |grep 'CA renewal master'
It came up blank. I suspect I didn't have a renewal master somehow.
Then I did This:
ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA
server)
Next is a "yum update" to be safe, and lastly "ipa-server-upgrade" on
ipa2. When that's all done I'll try "yum update" and
"ipa-server-upgrade" on my broken IPA system ipa1
I'll report back here when finished.
Thanks,
-Chris
Check to see which masteris the renewal master. Look in
cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
ipaConfigString=caRenewalMaster
You want to run the script on that master first to get the certs
renewed.
I'd start by re-running ipa-server-upgrade. It is idempotent so there
should be no risk. It may repair the tracking for you.
rob
On 2/4/2019 3:30 PM, Rob Crittenden wrote:
Chris Mohler via FreeIPA-users wrote:
Thanks for looking at my issue!
There have been no recent updates on my system. Actually I was getting
ready to update when I noticed things weren't good.
Here is the output from the log of the most recent update. Looks
like it
was completed successfully. The lines you asked about are in
Bold/underlined.
2018-07-18T16:55:21Z INFO [Update certmonger certificate renewal
configuration]
2018-07-18T16:55:21Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2018-07-18T16:55:21Z DEBUG Starting external process
2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d
/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt
2018-07-18T16:55:21Z DEBUG Process finished, return code=0
2018-07-18T16:55:21Z DEBUG stdout=
Certificate Nickname Trust
Attributes
SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca CTu,Cu,Cu
subsystemCert cert-pki-ca u,u,u
ocspSigningCert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
Server-Cert cert-pki-ca u,u,u
2018-07-18T16:55:21Z DEBUG stderr=
_*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop tracking
system certificates for CA*_
2018-07-18T16:55:21Z DEBUG Starting external process
2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
messagebus.service
2018-07-18T16:55:21Z DEBUG Process finished, return code=0
2018-07-18T16:55:21Z DEBUG stdout=
2018-07-18T16:55:21Z DEBUG stderr=
2018-07-18T16:55:21Z DEBUG Starting external process
2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active
messagebus.service
2018-07-18T16:55:21Z DEBUG Process finished, return code=0
2018-07-18T16:55:21Z DEBUG stdout=active
2018-07-18T16:55:21Z DEBUG stderr=
2018-07-18T16:55:21Z DEBUG Starting external process
2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
certmonger.service
2018-07-18T16:55:21Z DEBUG Process finished, return code=0
2018-07-18T16:55:21Z DEBUG stdout=
2018-07-18T16:55:21Z DEBUG stderr=
2018-07-18T16:55:21Z DEBUG Starting external process
2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active
certmonger.service
2018-07-18T16:55:21Z DEBUG Process finished, return code=0
2018-07-18T16:55:21Z DEBUG stdout=active
-snip- a few more lines like the section above.
2018-07-18T16:55:25Z DEBUG stderr=
2018-07-18T16:55:30Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2018-07-18T16:55:30Z DEBUG Starting external process
2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d
/etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f
/etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt
2018-07-18T16:55:30Z DEBUG Process finished, return code=0
2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN CERTIFICATE-----
-Snip- Cert and Key stuff goes here-
2018-07-18T16:55:34Z DEBUG stderr=
_*2018-07-18T16:55:35Z INFO Certmonger certificate renewal
configuration updated*_
Check to see which masteris the renewal master. Look in
cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
ipaConfigString=caRenewalMaster
You want to run the script on that master first to get the certs
renewed.
I'd start by re-running ipa-server-upgrade. It is idempotent so there
should be no risk. It may repair the tracking for you.
rob
On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote:
On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote:
Hi Everyone,
I'm looking for some help. I'm having trouble with everything
basically.
I think one of my CA's certs expired or something. I can't kinit
admin, I can't login via the WebGui. If I "getcert list" it returns
"Number of certificates and requests being tracked: 0."
This all started happening a few days ago and I am at a loss as to
what happened. On a whim I set the system date and time back a few
months to see if my certs were expired and like magic I can login to
the Webgui but I'm still not tracking anything with "getcert list" I
suspect the cert has expired but without tracking it I can't
tell, or
renew it.
Hi,
can you check if an upgrade happened recently (have a look at
/var/log/ipaupgrade.log)? The upgrade stop tracking certs and
re-configures certmonger, so if it failed in the middle you may be
left without any tracking.
You should be able to find lines like the following if the
untracking/tracking went fine:
---
[Update certmonger certificate renewal configuration]
Configuring certmonger to stop tracking system certificates for CA
Certmonger certificate renewal configuration updated
---
HTH,
flo
Please help
I'm running Centos 7, FreeIPA 4.5.4
Thanks,
-Chris
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to
[email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to
[email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[email protected]
