Hi! Forgot kinit:
-- kinit admin Password for admin@<<REALM>>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<<REALM>> Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<<ipa1.fqdn>>@<<REALM>> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<<REALM>>@<<REALM>> ipa config-show ipa: ERROR: No valid Negotiate header in server response -- Still no luck getting ipa config Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:[email protected]] Sent: torstai 21. kesäkuuta 2018 9.43 To: FreeIPA users list <[email protected]> Cc: Jokinen Eemeli <[email protected]> Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > > We do have 2 IPA nodes configured but the second node has been down for some > time. Tried to update it to same version as node1: > - Won't start tells me to use ipa-server-upgrade > - Ipa-server-upgrade fails at start, doesn't start directory server > - /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse > errors but last row tells me > -- > EMERG - main - Fatal Error---No ports specified. Exiting now. > -- > Hi, in this case let's concentrate on node1. > On node1 ipa config-show results only > -- > ipa: ERROR: did not receive Kerberos credentials > -- You need to run kinit to get kerberos credentials before any ipa * command. If the ipa stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show If node1 is not the renewal master, make it the renewal master from now on (follow instructions from https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/server-roles#server-roles-promote-to-ca). After this step, check the certificates in the NSS databases vs the certificates stored in LDAP. HTH, Flo > > > Eemeli > > -----Original Message----- > From: Florence Blanc-Renaud [mailto:[email protected]] > Sent: keskiviikko 20. kesäkuuta 2018 19.49 > To: FreeIPA users list <[email protected]> > Cc: Jokinen Eemeli <[email protected]> > Subject: Re: [Freeipa-users] Problems after IPA upgrade: > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > > On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: >> Hello all! >> >> I have very similiar problem as this one: >> >> https://lists.fedorahosted.org/archives/list/[email protected] >> r ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ >> >> ipa-server-upgrade fails as below >> >> -- >> >> Update complete >> >> Upgrading IPA services >> >> Upgrading the configuration of the IPA services >> >> [Verifying that root certificate is published] >> >> [Migrate CRL publish directory] >> >> CRL tree already moved >> >> [Verifying that CA proxy configuration is correct] >> >> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run >> command ipa-server-upgrade manually. >> >> CA did not start in 300.0s >> >> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log >> for more information >> >> -- >> >> And the log tells me that CA returns status 500 >> >> -- >> >> DEBUG Waiting for CA to start... >> >> DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus >> <http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus> >> >> DEBUG request body '' >> >> DEBUG response status 500 >> >> DEBUG response headers Server: Apache-Coyote/1.1 >> >> Content-Type: text/html;charset=utf-8 >> >> Content-Language: en >> >> Content-Length: 2208 >> >> Date: Fri, 15 Jun 2018 10:05:29 GMT >> >> Connection: close >> >> DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error >> report</title><style><!--H1 >> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 >> 5 >> D76;font-size:22px;} >> H2 >> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 >> 5 >> D76;font-size:16px;} >> H3 >> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 >> 5 >> D76;font-size:14px;} >> BODY >> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whi >> t >> e;} B >> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52 >> 5 >> D76;} >> P >> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;fon >> t -size:12px;}A {color : black;}A.name {color : black;}HR {color : >> #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem >> unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> >> Exception report</p><p><b>message</b> <u>Subsystem >> unavailable</u></p><p><b>description</b> <u>The server encountered an >> internal error that prevented it from fulfilling this >> request.</u></p><p><b>exception</b> >> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem >> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra >> i >> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent >> i >> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v >> a >> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache. >> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg. >> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44 >> 5 >> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac >> t >> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst >> r >> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache. >> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) >> \ >> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut >> o >> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th >> r >> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea >> d >> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr >> e ad.java:748)\n</pre></p><p><b>note</b> >> <u>The full stack trace of the root cause is available in the Apache >> Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache >> Tomcat/7.0.76</h3></body></html>' >> >> DEBUG The CA status is: check interrupted due to error: Retrieving CA >> status failed with status 500 >> >> DEBUG Waiting for CA to start... >> >> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and >> run command ipa-server-upgrade manually. >> >> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line >> 172, in execute >> >> return_value = self.run() >> >> File >> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad >> e >> .py", >> line 48, in run >> >> raise admintool.ScriptError(str(e)) >> >> The ipa-server-upgrade command failed, exception: ScriptError: CA did >> not start in 300.0s >> >> ERROR CA did not start in 300.0s >> >> -- >> >> With command "ipactl start --ignore-service-failures" I can start all >> the services but pki-tomcatd. >> >> -- >> >> Directory Service: RUNNING >> >> krb5kdc Service: RUNNING >> >> kadmin Service: RUNNING >> >> named Service: RUNNING >> >> httpd Service: RUNNING >> >> pki-tomcatd Service: STOPPED >> >> smb Service: RUNNING >> >> winbind Service: RUNNING >> >> ipa-otpd Service: RUNNING >> >> ipa-dnskeysyncd Service: RUNNING >> >> ipa: INFO: The ipactl command was successful >> >> -- >> >> Suggested resolution to above problem doesn't help me since the LDAP >> and NSS DB seem to have same certificates (some difference in >> wrapping but the string is same if I take out the line breaks) and >> even the serial number matches. >> >> -- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' -a >> >> -----BEGIN CERTIFICATE----- >> >> MIIDjD. >> >> .Prh2G >> >> -----END CERTIFICATE----- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' >> |grep Serial >> >> Serial Number: 4 (0x4) >> >> ldapsearch -LLL -D 'cn=directory manager' -W -b >> uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso >> >> Enter LDAP Password: >> >> dn: uid=pkidbuser,ou=people,o=ipaca >> >> userCertificate:: MIIDjD. >> >> .Prh2 >> >> G >> >> description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA >> Subsystem, >> >> O=<<REALM>> >> >> seeAlso: CN=CA Subsystem,O=<<REALM>> >> >> -- >> >> And here's where my actual knowledge of things end. I've been trying >> to figure out all kind of logs (tomcat, Kerberos, directory server, >> .) but haven't found a solid reason for it. I'm starting to believe >> this is a certificate issue, because although "getcert list" tells me >> that the certificate status is "Monitoring" on all certificates the >> expiry date is already in the past (current date 20.6.2018, >> certificate expiry >> 21.03.2018) on 4 certificates and it won't update even if I resubmit >> it or delete certificate and manually redo it (it got the same date >> as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", >> "Server-Cert cert-pki-ca" and two directory server certs) are valid >> for years (until >> 2020+). >> >> -- >> >> Request ID '20160331084234': >> >> status: MONITORING >> >> stuck: no >> >> key pair storage: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning >> C ert cert-pki-ca',token='NSS Certificate DB',pin set >> >> certificate: >> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning >> C ert cert-pki-ca',token='NSS Certificate DB' >> >> CA: dogtag-ipa-ca-renew-agent >> >> issuer: CN=Certificate Authority,O=<<REALM>> >> >> subject: CN=OCSP Subsystem,O=<<REALM>> >> >> expires: 2018-03-21 09:42:04 UTC >> >> key usage: >> digitalSignature,nonRepudiation,keyCertSign,cRLSign >> >> eku: id-kp-OCSPSigning >> >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad >> >> post-save command: >> /usr/libexec/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> >> track: yes >> >> auto-renew: yes >> >> Request ID '20160331085008': >> >> status: MONITORING >> >> stuck: no >> >> key pair storage: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' >> N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' >> >> certificate: >> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token=' >> N >> SS >> Certificate DB' >> >> CA: IPA >> >> issuer: CN=Certificate Authority,O=<<REALM>> >> >> subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>> >> >> expires: 2020-03-04 09:58:23 UTC >> >> principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>> >> >> key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >> eku: id-kp-serverAuth,id-kp-clientAuth >> >> pre-save command: >> >> post-save command: /usr/lib64/ipa/certmonger/restart_httpd >> >> track: yes >> >> auto-renew: yes >> >> -- >> >> Has anyone else bumped into same kind of issues? Any ideas where I >> should continue looking? I'm starting to run out of ideas. >> >> Eemeli Jokinen >> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- [email protected] >> To unsubscribe send an email to >> [email protected] >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] >> o rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ >> > > Hi, > > does your topology include multiple CA instances? You need first to find > which master is the CA renewal master: > ipa config-show | grep "renewal master" > > On this host, check that the certificates are still valid and consistent with > the content of the LDAP entries. If it is not the case, you need to repair > the CA renewal master first. > > When the CA renewal master is OK, check if the replication is working with > the other CA instances, and repair the other masters. > HTH, > Flo > _______________________________________________ > FreeIPA-users mailing list -- [email protected] > To unsubscribe send an email to > [email protected] > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/ > _______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]/message/7COMUEBZFYX6KV55HWWNFZ2H6JKDTNVB/
