On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus
<http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
172, in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did
not start in 300.0s
ERROR CA did not start in 300.0s
--
With command “ipactl start --ignore-service-failures” I can start all
the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn’t help me since the LDAP and
NSS DB seem to have same certificates (some difference in wrapping but
the string is same if I take out the line breaks) and even the serial
number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD…
…Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
|grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD…
…Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here’s where my actual knowledge of things end. I’ve been trying to
figure out all kind of logs (tomcat, Kerberos, directory server, …) but
haven’t found a solid reason for it. I’m starting to believe this is a
certificate issue, because although “getcert list” tells me that the
certificate status is “Monitoring” on all certificates the expiry date
is already in the past (current date 20.6.2018, certificate expiry
21.03.2018) on 4 certificates and it won’t update even if I resubmit it
or delete certificate and manually redo it (it got the same date as the
“old ones”). The “main certs” (“caSigningCert cert-pki-ca”, “Server-Cert
cert-pki-ca” and two directory server certs) are valid for years (until
2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I
should continue looking? I’m starting to run out of ideas…
Eemeli Jokinen
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find
which master is the CA renewal master:
ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent
with the content of the LDAP entries. If it is not the case, you need to
repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working
with the other CA instances, and repair the other masters.
HTH,
Flo
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/GYLYVYHEYFSSS4U5LNO7TSQERNT7VHO2/
