Hi!
I reply to this since there's some data in this message queue already related
to my problem:
I had 2 ipa node cluster, where the second node had been offline for some time
and at some point we had an error while trying to reboot node1 which was a
Renewal Master. The issue was that some certs had expired and after a bit of
special work we got the node1 back on track. I can spot three problems and I
can't (again) figure out which one is the cause and which one I should repair
first.
Now I got assigned the case to get the node2 back on track also. It had some
certificates expired (obviously) so I did a small time jump and some of the
certs were renewed. However not all of them were upgraded. "getcert list"
reports 3 certs "CA Unreachable", other 3 certs seem fine.
--
getcert list |grep -A 10 "CA_UNREACH"
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
--
status: CA_UNREACHABLE
ca-error: Internal error
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=IPA RA,O=<<REALM>>
expires: 2018-03-21 09:42:29 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
--
status: CA_UNREACHABLE
ca-error: Error 7 connecting to
http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipa2.fqdn>>,O=<<REALM>>
expires: 2018-06-27 07:01:38 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
--
Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) but
shouldn't node1 be the renewal master? Restarting httpd, certmonger and
pki-tomcat don't seem to help, time traveling helped on other certs but not on
these.
Directory service seems to work if I start it manually but ipa-server-upgrade
fails on directory server not starting with "No ports specified" so something
wrong with it or is it the certificates?
--
ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/10]: stopping directory server
[2/10]: saving configuration
[3/10]: disabling listeners
[4/10]: enabling DS global lock
[5/10]: starting directory server
--
<<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EMERG -
main - Fatal Error---No ports specified. Exiting now.
--
Also certmonger has issues:
--
dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last):
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line
541, in <module>
sys.exit(main())
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line
515, in main
kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line
43, in kinit_keytab
cred = gssapi.Credentials(name=name, store=store, usage='initiate')
File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in
__new__
store=store)
File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148,
in acquire
usage)
File "ext_cred_store.pyx", line 182, in
gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732)
GSSError: Major (851968): Unspecified GSS failure. Minor code may
provide more information, Minor (2529639068): Cannot contact any KDC for realm
'<<REALM>>'
--
but KDCs should be able to be resolved even from ipa node2
--
nslookup -type=srv _kerberos._tcp.<<REALM>>
Server: <<ipa1.ip>>
Address: <<ipa1.ip>>#53
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>.
_kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>.
--
For testing purposes I turned off firewall on ipa node1
Eemeli
-----Original Message-----
From: Rob Crittenden [mailto:rcrit...@redhat.com]
Sent: torstai 28. kesäkuuta 2018 16.05
To: Jokinen Eemeli <eemeli.joki...@cinia.fi>; FreeIPA users list
<freeipa-users@lists.fedorahosted.org>; Florence Blanc-Renaud <f...@redhat.com>
Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade
doesn't complete, pki-tomcatd won't start
Jokinen Eemeli wrote:
> Hi!
>
> No I haven't since my guide line didn't tell me to.
>
> I tried to set the date back, restart certmonger and then I did "ipactl
> restart" and then it got 2 certs renewed! One of the remaining two
> certificates was on "CA_UNREACHABLE" state, so I ran another certmonger
> restart and it did get updated. The last one didn't seem to go anywhere so I
> resubmitted the cert request and then that one also got renewed. I time
> jumped back to today and did another ipactl restart and All this mess got
> started with failed "ipa-server-upgrade" so I ran it afterwards and it
> completed successfully with no errors. That also increased the number of
> certmonger tracked certificates to 9 from 8 so I believe that one is fixed
> too.
There are layers of dependencies on the certs so sometimes multiple rounds of
renewal are needed to sort things out. This normally happens gracefully as
expiration approaches but in some cases that we haven't been able to identify
this doesn't happen.
>
> Thank you a lot! It's a bit complicated mess to understand every aspect of it
> (for example I was trying to hunt missing certificate that certmonger didn't
> track even though it wasn't the issue but the outcome of failed server
> upgrade) but after this I believe very that I understand it a way better!
Cool, glad you are back up and running.
Note that the cert issues weren't caused by the upgrade, the upgrade just made
it more apparent.
In order to be sure the upgrade is complete you should run: # ipa-server-upgrade
The upgrade will also check all of the certs tracked by certmonger and ensure
they are set up correctly.
rob
>
>
> Eemeli
>
> -----Original Message-----
> From: Rob Crittenden [mailto:rcrit...@redhat.com]
> Sent: keskiviikko 27. kesäkuuta 2018 16.26
> To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>;
> Florence Blanc-Renaud <f...@redhat.com>
> Cc: Jokinen Eemeli <eemeli.joki...@cinia.fi>
> Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade:
> ipa-server-upgrade doesn't complete, pki-tomcatd won't start
>
>
> Hard to know without seeing the list of certs.
>
> Are you restarting dogtag, Apache and 389-ds when setting the date back?
> That is necessary as well.
>
> rob
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/O6QB4Y32BK3EW44SUA664RRFMBJOKIH2/
