On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote:
Hi!

We do have 2 IPA nodes configured but the second node has been down for some 
time. Tried to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
-  /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse errors but 
last row tells me
--
EMERG - main - Fatal Error---No ports specified. Exiting now.
--

Hi,

in this case let's concentrate on node1.

On node1 ipa config-show results only
--
ipa: ERROR: did not receive Kerberos credentials
--

You need to run kinit to get kerberos credentials before any ipa * command. If the ipa stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring failed services:
$ sudo ipactl start --ignore-service-failures

Then to check who is renewal master:
$ sudo kinit admin
$ sudo ipa config-show

If node1 is not the renewal master, make it the renewal master from now on (follow instructions from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/server-roles#server-roles-promote-to-ca). After this step, check the certificates in the NSS databases vs the certificates stored in LDAP.

HTH,
Flo



Eemeli

-----Original Message-----
From: Florence Blanc-Renaud [mailto:[email protected]]
Sent: keskiviikko 20. kesäkuuta 2018 19.49
To: FreeIPA users list <[email protected]>
Cc: Jokinen Eemeli <[email protected]>
Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade 
doesn't complete, pki-tomcatd won't start

On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!

I have very similiar problem as this one:

https://lists.fedorahosted.org/archives/list/[email protected]
ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/

ipa-server-upgrade fails as below

--

Update complete

Upgrading IPA services

Upgrading the configuration of the IPA services

[Verifying that root certificate is published]

[Migrate CRL publish directory]

CRL tree already moved

[Verifying that CA proxy configuration is correct]

IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.

CA did not start in 300.0s

The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information

--

And the log tells me that CA returns status 500

--

DEBUG Waiting for CA to start...

DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus
<http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>

DEBUG request body ''

DEBUG response status 500

DEBUG response headers Server: Apache-Coyote/1.1

Content-Type: text/html;charset=utf-8

Content-Language: en

Content-Length: 2208

Date: Fri, 15 Jun 2018 10:05:29 GMT

Connection: close

DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:whit
e;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font
-size:12px;}A {color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai
nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti
catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va
lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445
)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr
actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\
n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr
eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread
$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre
ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'

DEBUG The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500

DEBUG Waiting for CA to start...

ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
run command ipa-server-upgrade manually.

File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
172, in execute

      return_value = self.run()

    File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade
.py",
line 48, in run

      raise admintool.ScriptError(str(e))

The ipa-server-upgrade command failed, exception: ScriptError: CA did
not start in 300.0s

ERROR CA did not start in 300.0s

--

With command "ipactl start --ignore-service-failures" I can start all
the services but pki-tomcatd.

--

Directory Service: RUNNING

krb5kdc Service: RUNNING

kadmin Service: RUNNING

named Service: RUNNING

httpd Service: RUNNING

pki-tomcatd Service: STOPPED

smb Service: RUNNING

winbind Service: RUNNING

ipa-otpd Service: RUNNING

ipa-dnskeysyncd Service: RUNNING

ipa: INFO: The ipactl command was successful

--

Suggested resolution to above problem doesn't help me since the LDAP
and NSS DB seem to have same certificates (some difference in wrapping
but the string is same if I take out the line breaks) and even the
serial number matches.

--

certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' -a

-----BEGIN CERTIFICATE-----

MIIDjD.

.Prh2G

-----END CERTIFICATE-----

certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
|grep Serial

          Serial Number: 4 (0x4)

ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso

Enter LDAP Password:

dn: uid=pkidbuser,ou=people,o=ipaca

userCertificate:: MIIDjD.

.Prh2

G

description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,

O=<<REALM>>

seeAlso: CN=CA Subsystem,O=<<REALM>>

--

And here's where my actual knowledge of things end. I've been trying
to figure out all kind of logs (tomcat, Kerberos, directory server, .)
but haven't found a solid reason for it. I'm starting to believe this
is a certificate issue, because although "getcert list" tells me that
the certificate status is "Monitoring" on all certificates the expiry
date is already in the past (current date 20.6.2018, certificate
expiry
21.03.2018) on 4 certificates and it won't update even if I resubmit
it or delete certificate and manually redo it (it got the same date as
the "old ones"). The "main certs" ("caSigningCert cert-pki-ca",
"Server-Cert cert-pki-ca" and two directory server certs) are valid
for years (until
2020+).

--

Request ID '20160331084234':

          status: MONITORING

          stuck: no

          key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
ert cert-pki-ca',token='NSS Certificate DB',pin set

          certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
ert cert-pki-ca',token='NSS Certificate DB'

          CA: dogtag-ipa-ca-renew-agent

          issuer: CN=Certificate Authority,O=<<REALM>>

          subject: CN=OCSP Subsystem,O=<<REALM>>

          expires: 2018-03-21 09:42:04 UTC

          key usage:
digitalSignature,nonRepudiation,keyCertSign,cRLSign

          eku: id-kp-OCSPSigning

          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

          post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"

          track: yes

          auto-renew: yes

Request ID '20160331085008':

          status: MONITORING

          stuck: no

          key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'

          certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
SS
Certificate DB'

          CA: IPA

          issuer: CN=Certificate Authority,O=<<REALM>>

          subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>

          expires: 2020-03-04 09:58:23 UTC

          principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>

          key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

          eku: id-kp-serverAuth,id-kp-clientAuth

          pre-save command:

          post-save command: /usr/lib64/ipa/certmonger/restart_httpd

          track: yes

          auto-renew: yes

--

Has anyone else bumped into same kind of issues? Any ideas where I
should continue looking? I'm starting to run out of ideas.

Eemeli Jokinen



_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to
[email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/


Hi,

does your topology include multiple CA instances? You need first to find which 
master is the CA renewal master:
ipa config-show | grep "renewal master"

On this host, check that the certificates are still valid and consistent with 
the content of the LDAP entries. If it is not the case, you need to repair the 
CA renewal master first.

When the CA renewal master is OK, check if the replication is working with the 
other CA instances, and repair the other masters.
HTH,
Flo
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/[email protected]/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/

_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/[email protected]/message/PKTXABYMSUI73K62YLF6XA2DFC6UVXNJ/

Reply via email to