Further attempts to fix the IPA server start has revealed that the ca
admin getStatus is returning a server error (500).
This has come up during restarts and ipa-server-upgrade.
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
<http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus>
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133',
'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
'close', 'date': 'Sat, 31 Dec 2016 18:44:55 GMT', 'content-type':
'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b> <pre>javax.ws.rs
<http://javax.ws.rs>.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
CA status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133',
'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
'close', 'date': 'Sat, 31 Dec 2016 18:44:56 GMT', 'content-type':
'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
CA status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA
server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
raise admintool.ScriptError(str(e))
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The
ipa-server-upgrade command failed, exception: ScriptError: CA did not
start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did
not start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The
ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information
with following in the syslog
Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase
backgroundProcess
WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@38406d47 background process
javax.ws.rs <http://javax.ws.rs>.ServiceUnavailableException: Subsystem
unavailable
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:745)
2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <[email protected]
<mailto:[email protected]>>:
Rob/Florence,
do you have any pointers on how to troubleshoot,
reinstall/configure, update or fix the PKI server to function properly?
Also if you know of any documentation or video that could be helpful.
I researched the typical suspects youtube and freeipa.org
<http://freeipa.org> without luck.
Daniel
2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl
<[email protected] <mailto:[email protected]>>:
I do not believe I changed the DM password. I know I had to
update the admin passwords regularly.
Only during the startup using ipactl start --force I am able to
connect to the service using the password for DM and it returns:
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#
#
dn:
objectClass: top
namingContexts: cn=changelog
namingContexts: dc=myorg,dc=com
namingContexts: o=ipaca
defaultnamingcontext: dc=myorg,dc=com
supportedExtension: 2.16.840.1.113730.3.5.7
supportedExtension: 2.16.840.1.113730.3.5.8
supportedExtension: 2.16.840.1.113730.3.5.10
supportedExtension: 2.16.840.1.113730.3.8.10.3
supportedExtension: 2.16.840.1.113730.3.8.10.4
supportedExtension: 2.16.840.1.113730.3.8.10.4.1
supportedExtension: 1.3.6.1.4.1.4203.1.11.1
supportedExtension: 2.16.840.1.113730.3.8.10.1
supportedExtension: 2.16.840.1.113730.3.8.10.5
supportedExtension: 2.16.840.1.113730.3.5.3
supportedExtension: 2.16.840.1.113730.3.5.12
supportedExtension: 2.16.840.1.113730.3.5.5
supportedExtension: 2.16.840.1.113730.3.5.6
supportedExtension: 2.16.840.1.113730.3.5.9
supportedExtension: 2.16.840.1.113730.3.5.4
supportedExtension: 2.16.840.1.113730.3.6.5
supportedExtension: 2.16.840.1.113730.3.6.6
supportedExtension: 2.16.840.1.113730.3.6.7
supportedExtension: 2.16.840.1.113730.3.6.8
supportedExtension: 1.3.6.1.4.1.1466.20037
supportedControl: 2.16.840.1.113730.3.4.2
supportedControl: 2.16.840.1.113730.3.4.3
supportedControl: 2.16.840.1.113730.3.4.4
supportedControl: 2.16.840.1.113730.3.4.5
supportedControl: 1.2.840.113556.1.4.473
supportedControl: 2.16.840.1.113730.3.4.9
supportedControl: 2.16.840.1.113730.3.4.16
supportedControl: 2.16.840.1.113730.3.4.15
supportedControl: 2.16.840.1.113730.3.4.17
supportedControl: 2.16.840.1.113730.3.4.19
supportedControl: 1.3.6.1.1.13.1
supportedControl: 1.3.6.1.1.13.2
supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
supportedControl: 1.2.840.113556.1.4.319
supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
supportedControl: 1.3.6.1.4.1.4203.666.5.16
supportedControl: 2.16.840.1.113730.3.8.10.6
supportedControl: 2.16.840.1.113730.3.4.14
supportedControl: 2.16.840.1.113730.3.4.20
supportedControl: 1.3.6.1.4.1.1466.29539.12
supportedControl: 2.16.840.1.113730.3.4.12
supportedControl: 2.16.840.1.113730.3.4.18
supportedControl: 2.16.840.1.113730.3.4.13
supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: DIGEST-MD5
supportedSASLMechanisms: CRAM-MD5
supportedSASLMechanisms: ANONYMOUS
supportedLDAPVersion: 2
supportedLDAPVersion: 3
vendorName: 389 Project
vendorVersion: 389-Directory/1.3.4.0 <http://1.3.4.0> B2016.215.1556
dataversion: 020161222235947020161222235947020161222235947
netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389
lastusn: 8690425
changeLog: cn=changelog
firstchangenumber: 2752153
lastchangenumber: 2752346
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
2016-12-21 9:27 GMT-06:00 Rob Crittenden <[email protected]
<mailto:[email protected]>>:
Daniel Schimpfoessl wrote:
> Thanks for getting back to me.
>
> getcert list | grep expires shows dates years in the
future for all
> certificates
> Inline-Bild 1
>
> ipactl start --force
>
> Eventually the system started with:
> Forced start, ignoring pki-tomcatd Service,
continuing normal
> operations.
>
> systemctl status ipa shows: failed
I don't think this is a certificate problem at all. I think
the timing
with your renewal is just coincidence.
Did you change your Directory Manager password at some point?
>
> ldapsearch -H ldaps://localhost:636 -D "cn=directory
manager" -w
> password -b "" -s base
> ldapsearch -H ldaps://localhost:636 -D "cn=directory
manager" -w
> *********** -b "" -s base
> Inline-Bild 2
You need the -x flag to indicate simple bind.
rob
> The logs have thousands of lines like it, what am I
looking for
> specifically?
>
> Daniel
>
>
> 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>:
>
> On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:
>
> Good day and happy holidays,
>
> I have been running a freeIPA instance for a few years
and been very
> happy. Recently the certificate expired and I updated it
using the
> documented methods. At first all seemed fine. Added a
Nagios
> monitor for
> the certificate expiration and restarted the server
(single
> server). I
> have weekly snapshots, daily backups (using Amanda on the
entire
> disk).
>
> One day the services relying on IPA failed to
authenticate.
> Looking at
> the server the ipa service had stopped. Restarting the
service
> fails.
> Restoring a few weeks old snapshot does not start either.
> Resetting the
> date to a few month back does not work either as httpd
fails to
> start .
>
> I am at a loss.
>
> Here a few details:
> # ipa --version
> VERSION: 4.4.0, API_VERSION: 2.213
>
>
> # /usr/sbin/ipactl start
> ...
> out -> Failed to start pki-tomcatd Service
> /var/log/pki/pki-tomcat/ca/debug -> Could not connect to
LDAP server
> host ipa.myorg.com <http://ipa.myorg.com>
<http://ipa.myorg.com> <http://ipa.myorg.com>
> port 636 Error
> netscape.ldap.LDAPException: Authentication failed (48)
> 2016-12-19T03:02:16Z DEBUG The CA status is: check
interrupted
> due to
> error: Retrieving CA status failed with status 500
>
> Any help would be appreciated as all connected services
are now
> down.
>
> Thanks,
>
> Daniel
>
>
>
>
> Hi Daniel,
>
> more information would be required to understand what
is going on.
> First of all, which certificate did you renew? Can you
check with
> $ getcert list
> if other certificates also expired?
>
> PKI fails to start and the error seems linked to the
SSL connection
> with the LDAP server. You may want to check if the
LDAP server is
> listening on the LDAPs port:
> - start the stack with
> $ ipactl start --force
> - check the LDAPs port with
> $ ldapsearch -H ldaps://localhost:636 -D "cn=directory
manager" -w
> password -b "" -s base
>
> The communication between PKI and the LDAP server is
authenticated
> with the certificate 'subsystemCert cert-pki-ca'
located in
> /etc/pki/pki-tomcat/alias, so you may also want to
check if it is
> still valid.
> The directory server access logs (in
> /var/log/dirsrv/slapd-DOMAIN-COM/access) would also
show the
> connection with logs similar to:
>
> [...] conn=47 fd=84 slot=84 SSL connection from
10.34.58.150 to
> 10.34.58.150
> [...] conn=47 TLS1.2 128-bit AES; client CN=CA
> Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>
<http://DOMAIN.COM>; issuer CN=Certificate
> Authority,O=DOMAIN.COM <http://DOMAIN.COM>
<http://DOMAIN.COM>
> [...] conn=47 TLS1.2 client bound as
uid=pkidbuser,ou=people,o=ipaca
> [...] conn=47 op=0 BIND dn="" method=sasl version=3
mech=EXTERNAL
> [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0
> dn="uid=pkidbuser,ou=people,o=ipaca"
>
>
>
> HTH,
> Flo
>
>
>
>
