Further attempts to fix the IPA server start has revealed that the ca admin getStatus is returning a server error (500).
This has come up during restarts and ipa-server-upgrade. ipa: DEBUG: Waiting for CA to start... ipa: DEBUG: request POST http://wwgwho01.webwim.com: 8080/ca/admin/ca/getStatus ipa: DEBUG: request body '' ipa: DEBUG: response status 500 ipa: DEBUG: response headers {'content-length': '2133', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat, 31 Dec 2016 18:44:55 GMT', 'content-type': 'text/html;charset=utf-8'} ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial, sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white; background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial, sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm. findSecurityConstraints(ProxyRealm.java:145)\n\torg. apache.catalina.authenticator.AuthenticatorBase.invoke( AuthenticatorBase.java:499)\n\torg.apache.catalina.valves. ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina. connector.CoyoteAdapter.service(CoyoteAdapter.java: 436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process( AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$ AbstractConnectionHandler.process(AbstractProtocol.java: 625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run( JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent. ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\ n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>' ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 ipa: DEBUG: Waiting for CA to start... ipa: DEBUG: request POST http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus ipa: DEBUG: request body '' ipa: DEBUG: response status 500 ipa: DEBUG: response headers {'content-length': '2133', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat, 31 Dec 2016 18:44:56 GMT', 'content-type': 'text/html;charset=utf-8'} ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>' ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 ipa: DEBUG: Waiting for CA to start... ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run raise admintool.ScriptError(str(e)) ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did not start in 300.0s ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information with following in the syslog Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase backgroundProcess WARNING: Exception processing realm com.netscape.cms.tomcat. ProxyRealm@38406d47 background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess( ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor. processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor. processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor. processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor. run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:745) 2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <[email protected]>: > Rob/Florence, > > do you have any pointers on how to troubleshoot, reinstall/configure, > update or fix the PKI server to function properly? > Also if you know of any documentation or video that could be helpful. > I researched the typical suspects youtube and freeipa.org without luck. > > Daniel > > 2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl <[email protected]>: > >> I do not believe I changed the DM password. I know I had to update the >> admin passwords regularly. >> >> Only during the startup using ipactl start --force I am able to connect >> to the service using the password for DM and it returns: >> >> # extended LDIF >> # >> # LDAPv3 >> # base <> with scope baseObject >> # filter: (objectclass=*) >> # requesting: ALL >> # >> >> # >> dn: >> objectClass: top >> namingContexts: cn=changelog >> namingContexts: dc=myorg,dc=com >> namingContexts: o=ipaca >> defaultnamingcontext: dc=myorg,dc=com >> supportedExtension: 2.16.840.1.113730.3.5.7 >> supportedExtension: 2.16.840.1.113730.3.5.8 >> supportedExtension: 2.16.840.1.113730.3.5.10 >> supportedExtension: 2.16.840.1.113730.3.8.10.3 >> supportedExtension: 2.16.840.1.113730.3.8.10.4 >> supportedExtension: 2.16.840.1.113730.3.8.10.4.1 >> supportedExtension: 1.3.6.1.4.1.4203.1.11.1 >> supportedExtension: 2.16.840.1.113730.3.8.10.1 >> supportedExtension: 2.16.840.1.113730.3.8.10.5 >> supportedExtension: 2.16.840.1.113730.3.5.3 >> supportedExtension: 2.16.840.1.113730.3.5.12 >> supportedExtension: 2.16.840.1.113730.3.5.5 >> supportedExtension: 2.16.840.1.113730.3.5.6 >> supportedExtension: 2.16.840.1.113730.3.5.9 >> supportedExtension: 2.16.840.1.113730.3.5.4 >> supportedExtension: 2.16.840.1.113730.3.6.5 >> supportedExtension: 2.16.840.1.113730.3.6.6 >> supportedExtension: 2.16.840.1.113730.3.6.7 >> supportedExtension: 2.16.840.1.113730.3.6.8 >> supportedExtension: 1.3.6.1.4.1.1466.20037 >> supportedControl: 2.16.840.1.113730.3.4.2 >> supportedControl: 2.16.840.1.113730.3.4.3 >> supportedControl: 2.16.840.1.113730.3.4.4 >> supportedControl: 2.16.840.1.113730.3.4.5 >> supportedControl: 1.2.840.113556.1.4.473 >> supportedControl: 2.16.840.1.113730.3.4.9 >> supportedControl: 2.16.840.1.113730.3.4.16 >> supportedControl: 2.16.840.1.113730.3.4.15 >> supportedControl: 2.16.840.1.113730.3.4.17 >> supportedControl: 2.16.840.1.113730.3.4.19 >> supportedControl: 1.3.6.1.1.13.1 >> supportedControl: 1.3.6.1.1.13.2 >> supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1 >> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2 >> supportedControl: 1.2.840.113556.1.4.319 >> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8 >> supportedControl: 1.3.6.1.4.1.4203.666.5.16 >> supportedControl: 2.16.840.1.113730.3.8.10.6 >> supportedControl: 2.16.840.1.113730.3.4.14 >> supportedControl: 2.16.840.1.113730.3.4.20 >> supportedControl: 1.3.6.1.4.1.1466.29539.12 >> supportedControl: 2.16.840.1.113730.3.4.12 >> supportedControl: 2.16.840.1.113730.3.4.18 >> supportedControl: 2.16.840.1.113730.3.4.13 >> supportedControl: 1.3.6.1.4.1.4203.1.9.1.1 >> supportedSASLMechanisms: EXTERNAL >> supportedSASLMechanisms: GSS-SPNEGO >> supportedSASLMechanisms: GSSAPI >> supportedSASLMechanisms: DIGEST-MD5 >> supportedSASLMechanisms: CRAM-MD5 >> supportedSASLMechanisms: ANONYMOUS >> supportedLDAPVersion: 2 >> supportedLDAPVersion: 3 >> vendorName: 389 Project >> vendorVersion: 389-Directory/1.3.4.0 B2016.215.1556 >> dataversion: 020161222235947020161222235947020161222235947 >> netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389 >> lastusn: 8690425 >> changeLog: cn=changelog >> firstchangenumber: 2752153 >> lastchangenumber: 2752346 >> >> # search result >> search: 2 >> result: 0 Success >> >> # numResponses: 2 >> # numEntries: 1 >> >> >> 2016-12-21 9:27 GMT-06:00 Rob Crittenden <[email protected]>: >> >>> Daniel Schimpfoessl wrote: >>> > Thanks for getting back to me. >>> > >>> > getcert list | grep expires shows dates years in the future for all >>> > certificates >>> > Inline-Bild 1 >>> > >>> > ipactl start --force >>> > >>> > Eventually the system started with: >>> > Forced start, ignoring pki-tomcatd Service, continuing normal >>> > operations. >>> > >>> > systemctl status ipa shows: failed >>> >>> I don't think this is a certificate problem at all. I think the timing >>> with your renewal is just coincidence. >>> >>> Did you change your Directory Manager password at some point? >>> >>> > >>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w >>> > password -b "" -s base >>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w >>> > *********** -b "" -s base >>> > Inline-Bild 2 >>> >>> You need the -x flag to indicate simple bind. >>> >>> rob >>> >>> > The logs have thousands of lines like it, what am I looking for >>> > specifically? >>> > >>> > Daniel >>> > >>> > >>> > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud <[email protected] >>> > <mailto:[email protected]>>: >>> > >>> > On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote: >>> > >>> > Good day and happy holidays, >>> > >>> > I have been running a freeIPA instance for a few years and >>> been very >>> > happy. Recently the certificate expired and I updated it using >>> the >>> > documented methods. At first all seemed fine. Added a Nagios >>> > monitor for >>> > the certificate expiration and restarted the server (single >>> > server). I >>> > have weekly snapshots, daily backups (using Amanda on the >>> entire >>> > disk). >>> > >>> > One day the services relying on IPA failed to authenticate. >>> > Looking at >>> > the server the ipa service had stopped. Restarting the service >>> > fails. >>> > Restoring a few weeks old snapshot does not start either. >>> > Resetting the >>> > date to a few month back does not work either as httpd fails to >>> > start . >>> > >>> > I am at a loss. >>> > >>> > Here a few details: >>> > # ipa --version >>> > VERSION: 4.4.0, API_VERSION: 2.213 >>> > >>> > >>> > # /usr/sbin/ipactl start >>> > ... >>> > out -> Failed to start pki-tomcatd Service >>> > /var/log/pki/pki-tomcat/ca/debug -> Could not connect to LDAP >>> server >>> > host ipa.myorg.com <http://ipa.myorg.com> < >>> http://ipa.myorg.com> >>> > port 636 Error >>> > netscape.ldap.LDAPException: Authentication failed (48) >>> > 2016-12-19T03:02:16Z DEBUG The CA status is: check interrupted >>> > due to >>> > error: Retrieving CA status failed with status 500 >>> > >>> > Any help would be appreciated as all connected services are now >>> > down. >>> > >>> > Thanks, >>> > >>> > Daniel >>> > >>> > >>> > >>> > >>> > Hi Daniel, >>> > >>> > more information would be required to understand what is going on. >>> > First of all, which certificate did you renew? Can you check with >>> > $ getcert list >>> > if other certificates also expired? >>> > >>> > PKI fails to start and the error seems linked to the SSL connection >>> > with the LDAP server. You may want to check if the LDAP server is >>> > listening on the LDAPs port: >>> > - start the stack with >>> > $ ipactl start --force >>> > - check the LDAPs port with >>> > $ ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w >>> > password -b "" -s base >>> > >>> > The communication between PKI and the LDAP server is authenticated >>> > with the certificate 'subsystemCert cert-pki-ca' located in >>> > /etc/pki/pki-tomcat/alias, so you may also want to check if it is >>> > still valid. >>> > The directory server access logs (in >>> > /var/log/dirsrv/slapd-DOMAIN-COM/access) would also show the >>> > connection with logs similar to: >>> > >>> > [...] conn=47 fd=84 slot=84 SSL connection from 10.34.58.150 to >>> > 10.34.58.150 >>> > [...] conn=47 TLS1.2 128-bit AES; client CN=CA >>> > Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>; issuer CN=Certificate >>> > Authority,O=DOMAIN.COM <http://DOMAIN.COM> >>> > [...] conn=47 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipac >>> a >>> > [...] conn=47 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL >>> > [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0 >>> > dn="uid=pkidbuser,ou=people,o=ipaca" >>> > >>> > >>> > >>> > HTH, >>> > Flo >>> > >>> > >>> > >>> > >>> >>> >> >
-- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
