On 12/31/2016 07:51 PM, Daniel Schimpfoessl wrote:
Further attempts to fix the IPA server start has revealed that the ca
admin getStatus is returning a server error (500).

This has come up during restarts and ipa-server-upgrade.

ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
<http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus>
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133',
'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
'close', 'date': 'Sat, 31 Dec 2016 18:44:55 GMT', 'content-type':
'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b> <pre>javax.ws.rs
<http://javax.ws.rs>.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
CA status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa: DEBUG: request POST
http://wwgwho01.webwim.com:8080/ca/admin/ca/getStatus
ipa: DEBUG: request body ''
ipa: DEBUG: response status 500
ipa: DEBUG: response headers {'content-length': '2133',
'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection':
'close', 'date': 'Sat, 31 Dec 2016 18:44:56 GMT', 'content-type':
'text/html;charset=utf-8'}
ipa: DEBUG: response body '<html><head><title>Apache Tomcat/7.0.69 -
Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR
size="1" noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.69</h3></body></html>'
ipa: DEBUG: The CA status is: check interrupted due to error: Retrieving
CA status failed with status 500
ipa: DEBUG: Waiting for CA to start...
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: IPA
server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG:   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 48, in run
    raise admintool.ScriptError(str(e))

ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: DEBUG: The
ipa-server-upgrade command failed, exception: ScriptError: CA did not
start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: CA did
not start in 300.0s
ipa.ipaserver.install.ipa_server_upgrade.ServerUpgrade: ERROR: The
ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
information


with following in the syslog
Dec 31, 2016 12:48:51 PM org.apache.catalina.core.ContainerBase
backgroundProcess
WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@38406d47 background process
javax.ws.rs <http://javax.ws.rs>.ServiceUnavailableException: Subsystem
unavailable
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:745)


2016-12-28 18:45 GMT-06:00 Daniel Schimpfoessl <dan...@schimpfoessl.com
<mailto:dan...@schimpfoessl.com>>:

    Rob/Florence,

    do you have any pointers on how to troubleshoot,
    reinstall/configure, update or fix the PKI server to function properly?
    Also if you know of any documentation or video that could be helpful.
    I researched the typical suspects youtube and freeipa.org
    <http://freeipa.org> without luck.

    Daniel

    2016-12-22 18:08 GMT-06:00 Daniel Schimpfoessl
    <dan...@schimpfoessl.com <mailto:dan...@schimpfoessl.com>>:

        I do not believe I changed the DM password. I know I had to
        update the admin passwords regularly.

        Only during the startup using ipactl start --force I am able to
        connect to the service using the password for DM and it returns:

        # extended LDIF
        #
        # LDAPv3
        # base <> with scope baseObject
        # filter: (objectclass=*)
        # requesting: ALL
        #

        #
        dn:
        objectClass: top
        namingContexts: cn=changelog
        namingContexts: dc=myorg,dc=com
        namingContexts: o=ipaca
        defaultnamingcontext: dc=myorg,dc=com
        supportedExtension: 2.16.840.1.113730.3.5.7
        supportedExtension: 2.16.840.1.113730.3.5.8
        supportedExtension: 2.16.840.1.113730.3.5.10
        supportedExtension: 2.16.840.1.113730.3.8.10.3
        supportedExtension: 2.16.840.1.113730.3.8.10.4
        supportedExtension: 2.16.840.1.113730.3.8.10.4.1
        supportedExtension: 1.3.6.1.4.1.4203.1.11.1
        supportedExtension: 2.16.840.1.113730.3.8.10.1
        supportedExtension: 2.16.840.1.113730.3.8.10.5
        supportedExtension: 2.16.840.1.113730.3.5.3
        supportedExtension: 2.16.840.1.113730.3.5.12
        supportedExtension: 2.16.840.1.113730.3.5.5
        supportedExtension: 2.16.840.1.113730.3.5.6
        supportedExtension: 2.16.840.1.113730.3.5.9
        supportedExtension: 2.16.840.1.113730.3.5.4
        supportedExtension: 2.16.840.1.113730.3.6.5
        supportedExtension: 2.16.840.1.113730.3.6.6
        supportedExtension: 2.16.840.1.113730.3.6.7
        supportedExtension: 2.16.840.1.113730.3.6.8
        supportedExtension: 1.3.6.1.4.1.1466.20037
        supportedControl: 2.16.840.1.113730.3.4.2
        supportedControl: 2.16.840.1.113730.3.4.3
        supportedControl: 2.16.840.1.113730.3.4.4
        supportedControl: 2.16.840.1.113730.3.4.5
        supportedControl: 1.2.840.113556.1.4.473
        supportedControl: 2.16.840.1.113730.3.4.9
        supportedControl: 2.16.840.1.113730.3.4.16
        supportedControl: 2.16.840.1.113730.3.4.15
        supportedControl: 2.16.840.1.113730.3.4.17
        supportedControl: 2.16.840.1.113730.3.4.19
        supportedControl: 1.3.6.1.1.13.1
        supportedControl: 1.3.6.1.1.13.2
        supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
        supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
        supportedControl: 1.2.840.113556.1.4.319
        supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
        supportedControl: 1.3.6.1.4.1.4203.666.5.16
        supportedControl: 2.16.840.1.113730.3.8.10.6
        supportedControl: 2.16.840.1.113730.3.4.14
        supportedControl: 2.16.840.1.113730.3.4.20
        supportedControl: 1.3.6.1.4.1.1466.29539.12
        supportedControl: 2.16.840.1.113730.3.4.12
        supportedControl: 2.16.840.1.113730.3.4.18
        supportedControl: 2.16.840.1.113730.3.4.13
        supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
        supportedSASLMechanisms: EXTERNAL
        supportedSASLMechanisms: GSS-SPNEGO
        supportedSASLMechanisms: GSSAPI
        supportedSASLMechanisms: DIGEST-MD5
        supportedSASLMechanisms: CRAM-MD5
        supportedSASLMechanisms: ANONYMOUS
        supportedLDAPVersion: 2
        supportedLDAPVersion: 3
        vendorName: 389 Project
        vendorVersion: 389-Directory/1.3.4.0 <http://1.3.4.0> B2016.215.1556
        dataversion: 020161222235947020161222235947020161222235947
        netscapemdsuffix: cn=ldap://dc=wwgwho01,dc=myorg,dc=com:389
        lastusn: 8690425
        changeLog: cn=changelog
        firstchangenumber: 2752153
        lastchangenumber: 2752346

        # search result
        search: 2
        result: 0 Success

        # numResponses: 2
        # numEntries: 1


        2016-12-21 9:27 GMT-06:00 Rob Crittenden <rcrit...@redhat.com
        <mailto:rcrit...@redhat.com>>:

            Daniel Schimpfoessl wrote:
            > Thanks for getting back to me.
            >
            > getcert list | grep expires shows dates years in the
            future for all
            > certificates
            > Inline-Bild 1
            >
            > ipactl start --force
            >
            > Eventually the system started with:
            >      Forced start, ignoring pki-tomcatd Service,
            continuing normal
            > operations.
            >
            > systemctl status ipa shows: failed

            I don't think this is a certificate problem at all. I think
            the timing
            with your renewal is just coincidence.

            Did you change your Directory Manager password at some point?

            >
            > ldapsearch -H ldaps://localhost:636 -D "cn=directory
            manager" -w
            > password -b "" -s base
            > ldapsearch -H ldaps://localhost:636 -D "cn=directory
            manager" -w
            > *********** -b "" -s base
            > Inline-Bild 2

            You need the -x flag to indicate simple bind.

            rob

            > The logs have thousands of lines like it, what am I
            looking for
            > specifically?
            >
            > Daniel
            >
            >
            > 2016-12-20 4:18 GMT-06:00 Florence Blanc-Renaud
            <f...@redhat.com <mailto:f...@redhat.com>
            > <mailto:f...@redhat.com <mailto:f...@redhat.com>>>:
            >
            >     On 12/19/2016 07:15 PM, Daniel Schimpfoessl wrote:
            >
            >         Good day and happy holidays,
            >
            >         I have been running a freeIPA instance for a few years 
and been very
            >         happy. Recently the certificate expired and I updated it 
using the
            >         documented methods. At first all seemed fine. Added a 
Nagios
            >         monitor for
            >         the certificate expiration and restarted the server 
(single
            >         server). I
            >         have weekly snapshots, daily backups (using Amanda on the 
entire
            >         disk).
            >
            >         One day the services relying on IPA failed to 
authenticate.
            >         Looking at
            >         the server the ipa service had stopped. Restarting the 
service
            >         fails.
            >         Restoring a few weeks old snapshot does not start either.
            >         Resetting the
            >         date to a few month back does not work either as httpd 
fails to
            >         start .
            >
            >         I am at a loss.
            >
            >         Here a few details:
            >         # ipa --version
            >         VERSION: 4.4.0, API_VERSION: 2.213
            >
            >
            >         # /usr/sbin/ipactl start
            >         ...
            >         out -> Failed to start pki-tomcatd Service
            >         /var/log/pki/pki-tomcat/ca/debug -> Could not connect to 
LDAP server
            >         host ipa.myorg.com <http://ipa.myorg.com>
            <http://ipa.myorg.com> <http://ipa.myorg.com>
            >         port 636 Error
            >         netscape.ldap.LDAPException: Authentication failed (48)
            >         2016-12-19T03:02:16Z DEBUG The CA status is: check 
interrupted
            >         due to
            >         error: Retrieving CA status failed with status 500
            >
            >         Any help would be appreciated as all connected services 
are now
            >         down.
            >
            >         Thanks,
            >
            >         Daniel
            >
            >
            >
            >
            >     Hi Daniel,
            >
            >     more information would be required to understand what
            is going on.
            >     First of all, which certificate did you renew? Can you
            check with
            >     $ getcert list
            >     if other certificates also expired?
            >
            >     PKI fails to start and the error seems linked to the
            SSL connection
            >     with the LDAP server. You may want to check if the
            LDAP server is
            >     listening on the LDAPs port:
            >     - start the stack with
            >     $ ipactl start --force
            >     - check the LDAPs port with
            >     $ ldapsearch -H ldaps://localhost:636 -D "cn=directory
            manager" -w
            >     password -b "" -s base
            >
            >     The communication between PKI and the LDAP server is
            authenticated
            >     with the certificate 'subsystemCert cert-pki-ca'
            located in
            >     /etc/pki/pki-tomcat/alias, so you may also want to
            check if it is
            >     still valid.
            >     The directory server access logs (in
            >     /var/log/dirsrv/slapd-DOMAIN-COM/access) would also
            show the
            >     connection with logs similar to:
            >
            >     [...] conn=47 fd=84 slot=84 SSL connection from
            10.34.58.150 to
            >     10.34.58.150
            >     [...] conn=47 TLS1.2 128-bit AES; client CN=CA
            >     Subsystem,O=DOMAIN.COM <http://DOMAIN.COM>
            <http://DOMAIN.COM>; issuer CN=Certificate
            >     Authority,O=DOMAIN.COM <http://DOMAIN.COM>
            <http://DOMAIN.COM>
            >     [...] conn=47 TLS1.2 client bound as
            uid=pkidbuser,ou=people,o=ipaca
            >     [...] conn=47 op=0 BIND dn="" method=sasl version=3
            mech=EXTERNAL
            >     [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0
            >     dn="uid=pkidbuser,ou=people,o=ipaca"
            >
            >
            >
            >     HTH,
            >     Flo
            >
            >
            >
            >




Hi Daniel,

the server error 500 means that PKI is not started. You can have a look at /var/log/pki/pki-tomcat/ca/debug, especially the logs generated when you try to start the service with
$ systemctl start pki-tomcatd@pki-tomcat.service

HTH,
Flo

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Reply via email to