On Sun, Jul 30, 2017 at 6:53 PM, Ian Harding via FreeIPA-users
<freeipa-users@lists.fedorahosted.org> wrote:
> I had an unexpected restart of an IPA server that had apparently had
> updates run but had not been restarted.  ipactl says pki-tomcatd would
> not start.
>
> Strangely, the actual service appears to be running:
>
> [root@seattlenfs slapd-BPT-ROCKS]# systemctl status
> pki-tomcatd@pki-tomcat.service
> ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
>    Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled;
> vendor preset: disabled)
>    Active: active (running) since Fri 2017-07-28 11:03:34 PDT; 36min ago
>   Process: 14289 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited,
> status=0/SUCCESS)
>  Main PID: 14406 (java)
>    CGroup:
> /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
>            └─14406 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
> -DRESTEASY_LIB=/usr/share/java/resteasy-base
> -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/...
>
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: Jul 28, 2017
> 11:39:50 AM org.apache.catalina.core.ContainerBase backgroundProcess
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: WARNING: Exception
> processing realm com.netscape.cms.tomcat.ProxyRealm@67cf2df background
> process
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]:
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
> Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at
> java.lang.Thread.run(Thread.java:748)
>
> However, the /var/log/ipaupgrade.log is full of trouble.  It ends with:
>
> 2017-07-28T17:01:19Z DEBUG The CA status is: check interrupted due to
> error: Retrieving CA status failed with status 500
> 2017-07-28T17:01:19Z DEBUG Waiting for CA to start...
> 2017-07-28T17:01:20Z DEBUG request POST
> http://seattlenfs.bpt.rocks:8080/ca/admin/ca/getStatus
> 2017-07-28T17:01:20Z DEBUG request body ''
> 2017-07-28T17:01:20Z DEBUG response status 500
> 2017-07-28T17:01:20Z DEBUG response headers {'content-length': '2208',
> 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
> 'close', 'date': 'Fri, 28 Jul 2017 17:01:20 GMT', 'content-type':
> 'text/html;charset=utf-8'}
> 2017-07-28T17:01:20Z DEBUG response body '<html><head><title>Apache
> Tomcat/7.0.69 - Error report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
> B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
> size="1" noshade="noshade"><p><b>type</b> Exception
> report</p><p><b>message</b> <u>Subsystem
> unavailable</u></p><p><b>description</b> <u>The server encountered an
> internal error that prevented it from fulfilling this
> request.</u></p><p><b>exception</b>
> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
> Tomcat/7.0.69</h3></body></html>'
> 2017-07-28T17:01:20Z DEBUG The CA status is: check interrupted due to
> error: Retrieving CA status failed with status 500
> 2017-07-28T17:01:20Z DEBUG Waiting for CA to start...
> 2017-07-28T17:01:21Z ERROR IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
> 2017-07-28T17:01:21Z DEBUG   File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
> execute
>     return_value = self.run()
>   File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> line 48, in run
>     raise admintool.ScriptError(str(e))
>
> 2017-07-28T17:01:21Z DEBUG The ipa-server-upgrade command failed,
> exception: ScriptError: CA did not start in 300.0s
> 2017-07-28T17:01:21Z ERROR CA did not start in 300.0s
> 2017-07-28T17:01:21Z ERROR The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>
>
> Should I just blindly run ipa-server-upgrade again?
>
> Googling had me look at certificate expirations, they seem to be good.
>
> [root@seattlenfs slapd-BPT-ROCKS]# getcert list | grep expires
>         expires: 2019-05-29 05:54:06 UTC
>         expires: 2019-05-29 05:53:57 UTC
>         expires: 2019-05-29 05:53:16 UTC
>         expires: 2035-07-16 12:51:42 UTC
>         expires: 2019-05-29 05:53:37 UTC
>         expires: 2018-08-15 05:20:24 UTC
>         expires: 2018-08-26 05:01:42 UTC
>         expires: 2018-08-26 05:01:43 UTC
>
> [root@seattlenfs slapd-BPT-ROCKS]# yum list | grep ipa-
> ipa-admintools.noarch                    4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-client.x86_64                        4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-client-common.noarch                 4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-common.noarch                        4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-python-compat.noarch                 4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-server.x86_64                        4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-server-common.noarch                 4.4.0-14.el7.centos.7
> @test-centos7-updates
> ipa-server-dns.noarch                    4.4.0-14.el7.centos.7
> @test-centos7-updates
>
> [root@seattlenfs slapd-BPT-ROCKS]# yum list | grep pki-
> pki-base.noarch                          10.3.3-19.el7_3
> @updates
> pki-base-java.noarch                     10.3.3-19.el7_3
> @updates
> pki-ca.noarch                            10.3.3-19.el7_3
> @updates
> pki-kra.noarch                           10.3.3-19.el7_3
> @updates
> pki-server.noarch                        10.3.3-19.el7_3
> @updates
> pki-tools.x86_64                         10.3.3-19.el7_3
> @updates
>
> [root@seattlenfs slapd-BPT-ROCKS]# yum list | grep tomcat
> tomcat.noarch                            7.0.69-12.el7_3
> @updates
> tomcat-el-2.2-api.noarch                 7.0.69-12.el7_3
> @updates
> tomcat-jsp-2.2-api.noarch                7.0.69-12.el7_3
> @updates
> tomcat-lib.noarch                        7.0.69-12.el7_3
> @updates
> tomcat-servlet-3.0-api.noarch            7.0.69-12.el7_3
> @updates
> tomcatjss.noarch                         7.1.2-3.el7
> @base
>
> [root@seattlenfs slapd-BPT-ROCKS]# yum list | grep java
> java-1.7.0-openjdk.x86_64                1:1.7.0.141-2.6.10.1.el7_3
> @test-centos7-updates
> java-1.7.0-openjdk-devel.x86_64          1:1.7.0.141-2.6.10.1.el7_3
> @test-centos7-updates
> java-1.7.0-openjdk-headless.x86_64       1:1.7.0.141-2.6.10.1.el7_3
> @test-centos7-updates
> java-1.8.0-openjdk.x86_64                1:1.8.0.141-1.b16.el7_3
> @updates
> java-1.8.0-openjdk-headless.x86_64       1:1.8.0.141-1.b16.el7_3
> @updates
> javamail.noarch                          1.4.6-8.el7
> @base
> javapackages-tools.noarch                3.4.1-11.el7
> @base
> javassist.noarch                         3.16.1-10.el7
> @base
> nuxwdog-client-java.x86_64               1.0.3-5.el7
> @base
> pki-base-java.noarch                     10.3.3-19.el7_3
> @updates
> python-javapackages.noarch               3.4.1-11.el7
> @base
> tzdata-java.noarch                       2017a-1.el7
> @test-centos7-updates
>
> Any other useful information I can provide?

There is a good page for reporting issues:

https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Dogtag_CA_failed

In your case, Dogtag CA did to start so the relevant section is:

ausearch -m AVC > avc.log
journalctl -u pki-tomcatd@pki-tomcat.service
/var/log/pki/pki-tomcat/ca/debug      <---- usually most important

additionally:
/var/log/pki/pki-tomcat/ca/selftests.log


only in install:
/var/log/pki/pki-ca-spawn.<latest>.log


-- 
Petr Vobornik
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org

Reply via email to