Further attempts to fix the IPA server start has revealed that the ca admin
getStatus is returning a server error (500).

This has come up during restarts and ipa-server-upgrade.

ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST http://wwgwho01.webwim.com:
8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133', 'content-language':
'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat,
31 Dec 2016 18:44:55 GMT', 'content-type': 'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error
report</title><style><!--H1 {font-family:Tahoma,Arial,
sans-serif;color:white;background-color:#525D76;font-size:22px;} H2
{font-family:Tahoma,Arial,sans-serif;color:white;
background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,
sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P 
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
<u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException:
Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.
findSecurityConstraints(ProxyRealm.java:145)\n\torg.
apache.catalina.authenticator.AuthenticatorBase.invoke(
AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.
ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.
connector.CoyoteAdapter.service(CoyoteAdapter.java:
436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(
AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$
AbstractConnectionHandler.process(AbstractProtocol.java:
625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(
JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.
ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\
n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133', 'content-language':
'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Sat,
31 Dec 2016 18:44:56 GMT', 'content-type': 'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
<u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA server
upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG:   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
    raise admintool.ScriptError(str(e))

ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The
ipa-server-upgrade command failed, exception: ScriptError: CA did not start
in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did not
start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The
ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information


with following in the syslog
Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase
backgroundProcess
WARNING: Exception processing realm com.netscape.cms.tomcat.
ProxyRealm@38406d47 background process
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
at org.apache.catalina.core.ContainerBase.backgroundProcess(
ContainerBase.java:1357)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1543)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.
run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:745)


2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <dan...@schimpfoessl.com>:

> Rob/Florence,
>
> do you have any pointers on how to troubleshoot, reinstall/configure,
> update or fix the PKI server to function properly?
> Also if you know of any documentation or video that could be helpful.
> I researched the typical suspects youtube and freeipa.org without luck.
>
> Daniel
>
> 2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl <dan...@schimpfoessl.com>:
>
>> I do not believe I changed the DM password. I know I had to update the
>> admin passwords regularly.
>>
>> Only during the startup using ipactl start --force I am able to connect
>> to the service using the password for DM and it returns:
>>
>> # extended LDIF
>> #
>> # LDAPv3
>> # base <> with scope baseObject
>> # filter: (objectclass=*)
>> # requesting: ALL
>> #
>>
>> #
>> dn:
>> objectClass: top
>> namingContexts: cn=changelog
>> namingContexts: dc=myorg,dc=com
>> namingContexts: o=ipaca
>> defaultnamingcontext: dc=myorg,dc=com
>> supportedExtension: 2.16.840.1.113730.3.5.7
>> supportedExtension: 2.16.840.1.113730.3.5.8
>> supportedExtension: 2.16.840.1.113730.3.5.10
>> supportedExtension: 2.16.840.1.113730.3.8.10.3
>> supportedExtension: 2.16.840.1.113730.3.8.10.4
>> supportedExtension: 2.16.840.1.113730.3.8.10.4.1
>> supportedExtension: 1.3.6.1.4.1.4203.1.11.1
>> supportedExtension: 2.16.840.1.113730.3.8.10.1
>> supportedExtension: 2.16.840.1.113730.3.8.10.5
>> supportedExtension: 2.16.840.1.113730.3.5.3
>> supportedExtension: 2.16.840.1.113730.3.5.12
>> supportedExtension: 2.16.840.1.113730.3.5.5
>> supportedExtension: 2.16.840.1.113730.3.5.6
>> supportedExtension: 2.16.840.1.113730.3.5.9
>> supportedExtension: 2.16.840.1.113730.3.5.4
>> supportedExtension: 2.16.840.1.113730.3.6.5
>> supportedExtension: 2.16.840.1.113730.3.6.6
>> supportedExtension: 2.16.840.1.113730.3.6.7
>> supportedExtension: 2.16.840.1.113730.3.6.8
>> supportedExtension: 1.3.6.1.4.1.1466.20037
>> supportedControl: 2.16.840.1.113730.3.4.2
>> supportedControl: 2.16.840.1.113730.3.4.3
>> supportedControl: 2.16.840.1.113730.3.4.4
>> supportedControl: 2.16.840.1.113730.3.4.5
>> supportedControl: 1.2.840.113556.1.4.473
>> supportedControl: 2.16.840.1.113730.3.4.9
>> supportedControl: 2.16.840.1.113730.3.4.16
>> supportedControl: 2.16.840.1.113730.3.4.15
>> supportedControl: 2.16.840.1.113730.3.4.17
>> supportedControl: 2.16.840.1.113730.3.4.19
>> supportedControl: 1.3.6.1.1.13.1
>> supportedControl: 1.3.6.1.1.13.2
>> supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
>> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
>> supportedControl: 1.2.840.113556.1.4.319
>> supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
>> supportedControl: 1.3.6.1.4.1.4203.666.5.16
>> supportedControl: 2.16.840.1.113730.3.8.10.6
>> supportedControl: 2.16.840.1.113730.3.4.14
>> supportedControl: 2.16.840.1.113730.3.4.20
>> supportedControl: 1.3.6.1.4.1.1466.29539.12
>> supportedControl: 2.16.840.1.113730.3.4.12
>> supportedControl: 2.16.840.1.113730.3.4.18
>> supportedControl: 2.16.840.1.113730.3.4.13
>> supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
>> supportedSASLMechanisms: EXTERNAL
>> supportedSASLMechanisms: GSS-SPNEGO
>> supportedSASLMechanisms: GSSAPI
>> supportedSASLMechanisms: DIGEST-MD5
>> supportedSASLMechanisms: CRAM-MD5
>> supportedSASLMechanisms: ANONYMOUS
>> supportedLDAPVersion: 2
>> supportedLDAPVersion: 3
>> vendorName: 389 Project
>> vendorVersion: 389-Directory/1.3.4.0 B2016.215.1556
>> dataversion: 020161222235947020161222235947020161222235947
>> netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389
>> lastusn: 8690425
>> changeLog: cn=changelog
>> firstchangenumber: 2752153
>> lastchangenumber: 2752346
>>
>> # search result
>> search: 2
>> result: 0 Success
>>
>> # numResponses: 2
>> # numEntries: 1
>>
>>
>> 2016-12-21 9:27 GMT-06:00 Rob Crittenden <rcrit...@redhat.com>:
>>
>>> Daniel Schimpfoessl wrote:
>>> > Thanks for getting back to me.
>>> >
>>> > getcert list | grep expires shows dates years in the future for all
>>> > certificates
>>> > Inline-Bild 1
>>> >
>>> > ipactl start --force
>>> >
>>> > Eventually the system started with:
>>> >      Forced start, ignoring pki-tomcatd Service, continuing normal
>>> > operations.
>>> >
>>> > systemctl status ipa shows: failed
>>>
>>> I don't think this is a certificate problem at all. I think the timing
>>> with your renewal is just coincidence.
>>>
>>> Did you change your Directory Manager password at some point?
>>>
>>> >
>>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> > password -b "" -s base
>>> > ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> > *********** -b "" -s base
>>> > Inline-Bild 2
>>>
>>> You need the -x flag to indicate simple bind.
>>>
>>> rob
>>>
>>> > The logs have thousands of lines like it, what am I looking for
>>> > specifically?
>>> >
>>> > Daniel
>>> >
>>> >
>>> > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud <f...@redhat.com
>>> > <mailto:f...@redhat.com>>:
>>> >
>>> >     On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:
>>> >
>>> >         Good day and happy holidays,
>>> >
>>> >         I have been running a freeIPA instance for a few years and
>>> been very
>>> >         happy. Recently the certificate expired and I updated it using
>>> the
>>> >         documented methods. At first all seemed fine. Added a Nagios
>>> >         monitor for
>>> >         the certificate expiration and restarted the server (single
>>> >         server). I
>>> >         have weekly snapshots, daily backups (using Amanda on the
>>> entire
>>> >         disk).
>>> >
>>> >         One day the services relying on IPA failed to authenticate.
>>> >         Looking at
>>> >         the server the ipa service had stopped. Restarting the service
>>> >         fails.
>>> >         Restoring a few weeks old snapshot does not start either.
>>> >         Resetting the
>>> >         date to a few month back does not work either as httpd fails to
>>> >         start .
>>> >
>>> >         I am at a loss.
>>> >
>>> >         Here a few details:
>>> >         # ipa --version
>>> >         VERSION: 4.4.0, API_VERSION: 2.213
>>> >
>>> >
>>> >         # /usr/sbin/ipactl start
>>> >         ...
>>> >         out -> Failed to start pki-tomcatd Service
>>> >         /var/log/pki/pki-tomcat/ca/debug -> Could not connect to LDAP
>>> server
>>> >         host ipa.myorg.com <http://ipa.myorg.com> <
>>> http://ipa.myorg.com>
>>> >         port 636 Error
>>> >         netscape.ldap.LDAPException: Authentication failed (48)
>>> >         2016-12-19T03:02:16Z DEBUG The CA status is: check interrupted
>>> >         due to
>>> >         error: Retrieving CA status failed with status 500
>>> >
>>> >         Any help would be appreciated as all connected services are now
>>> >         down.
>>> >
>>> >         Thanks,
>>> >
>>> >         Daniel
>>> >
>>> >
>>> >
>>> >
>>> >     Hi Daniel,
>>> >
>>> >     more information would be required to understand what is going on.
>>> >     First of all, which certificate did you renew? Can you check with
>>> >     $ getcert list
>>> >     if other certificates also expired?
>>> >
>>> >     PKI fails to start and the error seems linked to the SSL connection
>>> >     with the LDAP server. You may want to check if the LDAP server is
>>> >     listening on the LDAPs port:
>>> >     - start the stack with
>>> >     $ ipactl start --force
>>> >     - check the LDAPs port with
>>> >     $ ldapsearch -H ldaps://localhost:636 -D "cn=directory manager" -w
>>> >     password -b "" -s base
>>> >
>>> >     The communication between PKI and the LDAP server is authenticated
>>> >     with the certificate 'subsystemCert cert-pki-ca' located in
>>> >     /etc/pki/pki-tomcat/alias, so you may also want to check if it is
>>> >     still valid.
>>> >     The directory server access logs (in
>>> >     /var/log/dirsrv/slapd-DOMAIN-COM/access) would also show the
>>> >     connection with logs similar to:
>>> >
>>> >     [...] conn=47 fd=84 slot=84 SSL connection from 10.34.58.150 to
>>> >     10.34.58.150
>>> >     [...] conn=47 TLS1.2 128-bit AES; client CN=CA
>>> >     Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>; issuer CN=Certificate
>>> >     Authority,O=DOMAIN.COM <http://DOMAIN.COM>
>>> >     [...] conn=47 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipac
>>> a
>>> >     [...] conn=47 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL
>>> >     [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0
>>> >     dn="uid=pkidbuser,ou=people,o=ipaca"
>>> >
>>> >
>>> >
>>> >     HTH,
>>> >     Flo
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to