Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]
- ERR - slapi_ldap_bind - Error: could not send startTLS request:
error -1 (Can't contact LDAP server) errno 107 (Transport endpoint is
not connected)
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]
- ERR - set_krb5_creds - Could not get initial credentials for
principal [ldap/[email protected]] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials
available (default cache: /tmp/krb5cc_389))
Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]
- ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
Manager
masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config]
authentication mechanism [SIMPLE]: error 32 (No such object)
Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:47:40 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:40 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:47:50 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:47:50 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:00 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:00 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:10 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:10 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize
credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any
KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP
connection.
Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize
credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact any
KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted LDAP
connection.
Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:20 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:20 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
Feb 4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable
Feb 4 17:48:30 ipa2 server: at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Feb 4 17:48:30 ipa2 server: at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Feb 4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748)
Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
132.162.1.131 port 67 (xid=0x27e7db13)
^C
[root@ipa2 log]# less /var/log/ipaupgrade.log
<p><b>note</b> <u>The full stack trace of the root cause is available
in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:13Z DEBUG Waiting for CA to start...
2019-02-04T22:46:14Z DEBUG request POST
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
2019-02-04T22:46:14Z DEBUG request body ''
2019-02-04T22:46:14Z DEBUG response status 500
2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Mon, 04 Feb 2019 22:46:14 GMT
Connection: close
2019-02-04T22:46:14Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:14Z DEBUG Waiting for CA to start...
2019-02-04T22:46:15Z DEBUG request POST
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
2019-02-04T22:46:15Z DEBUG request body ''
2019-02-04T22:46:15Z DEBUG response status 500
2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Mon, 04 Feb 2019 22:46:15 GMT
Connection: close
2019-02-04T22:46:15Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to
error: Retrieving CA status failed with status 500
2019-02-04T22:46:15Z DEBUG Waiting for CA to start...
2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-02-04T22:46:16Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178,
in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 56, in run
raise admintool.ScriptError(str(e))
2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed,
exception: ScriptError: CA did not start in 300.0s
2019-02-04T22:46:16Z ERROR CA did not start in 300.0s
2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information