Hi!
Date: 20-03-2018
Services running (certmonger, dirsrv, httpd, pki-tomcatd)
--
ipa-getcert resubmit -i 20170425122557
Resubmitting "20170425122557" to "dogtag-ipa-ca-renew-agent".
getcert list |grep -A 1 20170425122557
Request ID '20170425122557':
status: CA_UNREACHABLE
--
Certmonger tells me:
--
Mar 20 03:26:02 <<ipa2.fqdn>> dogtag-ipa-ca-renew-agent-submit[14559]:
Traceback (most recent call last):
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line
541, in <module>
sys.exit(main())
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line
517, in main
api.Backend.ldap2.connect()
File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66,
in connect
conn = self.create_connection(*args, **kw)
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py",
line 190, in create_connection
client_controls=clientctrls)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line
1111, in external_bind
'', auth_tokens, server_controls, client_controls)
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line
1005, in error_handler
error=info)
NetworkError: cannot connect to
'ldapi://%2fvar%2frun%2fslapd-<<REALM>>.socket':
Mar 20 03:26:02 <<ipa2.fqdn>> certmonger[14121]: 2018-03-20 03:26:02 [14121]
Internal error
--
Next?
Eemeli
-----Original Message-----
From: Florence Blanc-Renaud [mailto:[email protected]]
Sent: perjantai 17. elokuuta 2018 14.43
To: FreeIPA users list <[email protected]>; Rob Crittenden
<[email protected]>
Cc: Jokinen Eemeli <[email protected]>
Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade
doesn't complete, pki-tomcatd won't start
On 08/17/2018 12:59 PM, Jokinen Eemeli via FreeIPA-users wrote:
> Hi!
>
> Yes, seems like there was "security: off" but that doesn't seem to do it, I
> think I have ended up in the situation that I need to recreate some
> certificates, because:
>
> I check the renewal dates.
>
> --
> getcert list |grep expires:
> expires: 2018-03-21 09:42:04 UTC
> expires: 2036-03-31 08:42:02 UTC
> expires: 2018-03-21 09:42:29 UTC
> expires: 2018-06-27 07:01:38 UTC
> expires: 2020-08-17 10:17:32 UTC
> expires: 2020-06-28 05:49:50 UTC
> --
>
> I timejump to "before oldest expired" = 2018-03-20. Dirsrv seems to start ok.
> Certmonger restarts ok.
>
> Httpd does not start. Error from /etc/httpd/logs/error_log:
>
> --
> [Tue Mar 20 07:44:39.500363 2018] [:warn] [pid 11961] NSSSessionCacheTimeout
> is deprecated. Ignoring.
> [Tue Mar 20 07:44:39.688595 2018] [:error] [pid 11961] SSL Library
> Error: -8181 Certificate has expired [Tue Mar 20 07:44:39.688637 2018]
> [:error] [pid 11961] Unable to verify certificate 'Server-Cert'. Add
> "NSSEnforceValidCerts off" to nss.conf so the server can start until the
> problem can be resolved.
> --
>
> Seems like httpd has managed to renew some certificates at some point:
>
> --
> certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep Not
> Not Before: Thu Jun 28 05:49:50 2018
> Not After : Sun Jun 28 05:49:50 2020
> --
>
> Should I remove httpd certificate to be able to start httpd in "before
> 21-03-2018? I can't seem to be able to renew these 2 (ipaCert,
> ocspSigningCert) without httpd because if I try to resubmit them I get
> "CA_Unreachable"?
>
You can temporarily allow httpd to start even with expired/not yet valid
certificates: edit /etc/httpd/conf.d/nss.conf and set the NSSEnforceValidCerts
parameter to off, then restart httpd. (Do not forget to revert the setting when
you have fixed everything). See [1] for more information.
HTH,
flo
[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/expired-certs
>
>
> Eemeli
>
> -----Original Message-----
> From: Florence Blanc-Renaud [mailto:[email protected]]
> Sent: torstai 16. elokuuta 2018 21.54
> To: FreeIPA users list <[email protected]>; Rob
> Crittenden <[email protected]>
> Cc: Jokinen Eemeli <[email protected]>
> Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade:
> ipa-server-upgrade doesn't complete, pki-tomcatd won't start
>
> On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote:
>> Hi!
>>
>> Anybody can help me with this one?
>>
>> Summary:
>>
>> 2 node freeipa server cluster, node 2 was initially down for other reasons
>> and node 1 (renewal master) had forgot to update own certificates which
>> resulted faulty cluster. With help from mailing list we got the node 1 back
>> online and it's working great! Now I'm trying to get node2 back to working
>> order in cluster but it won't update the certificates even when trying the
>> timejump. Seems like it tries to renew certificates locally although somehow
>> I tought that it should renew the certificates from node 1...?
>>
> Hi,
>
> you probably have a combination of multiple issues on your second node.
>
> The ipa-server-upgrade failure may leave your instance in a wrong
> state, where dse.ldif has disabled the ports for 389-ds (see BZ
> https://bugzilla.redhat.com/show_bug.cgi?id=1569011 or pagure ticket
> https://pagure.io/freeipa/issue/7534).
> During the upgrade, dse.ldif is edited in order to temporarily disable the
> LDAP ports (to prevent ldap modifications during the upgrade).
> Sometimes, if the upgrade fails, dse.ldif is not restored and the ports
> remain disabled. You will have to stop the ldap server, manually edit
> dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set:
> nsslapd-port: 389
> nsslapd-security: on
>
> then restart the LDAP server.
>
> For the cert renewal, your procedure is the valid one. The kerberos error is
> probably linked to 389-ds not being accessible.
>
> HTH,
> flo
>>
>> Eemeli
>>
>> -----Original Message-----
>> From: Jokinen Eemeli
>> Sent: keskiviikko 4. heinäkuuta 2018 16.08
>> To: 'Rob Crittenden' <[email protected]>; FreeIPA users list
>> <[email protected]>; Florence Blanc-Renaud
>> <[email protected]>
>> Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade:
>> ipa-server-upgrade doesn't complete, pki-tomcatd won't start
>>
>> Hi!
>>
>> I reply to this since there's some data in this message queue already
>> related to my problem:
>>
>> I had 2 ipa node cluster, where the second node had been offline for some
>> time and at some point we had an error while trying to reboot node1 which
>> was a Renewal Master. The issue was that some certs had expired and after a
>> bit of special work we got the node1 back on track. I can spot three
>> problems and I can't (again) figure out which one is the cause and which one
>> I should repair first.
>>
>> Now I got assigned the case to get the node2 back on track also. It had some
>> certificates expired (obviously) so I did a small time jump and some of the
>> certs were renewed. However not all of them were upgraded. "getcert list"
>> reports 3 certs "CA Unreachable", other 3 certs seem fine.
>>
>> --
>> getcert list |grep -A 10 "CA_UNREACH"
>> status: CA_UNREACHABLE
>> ca-error: Internal error
>> stuck: no
>> key pair storage:
>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=Certificate Authority,O=<<REALM>>
>> subject: CN=OCSP Subsystem,O=<<REALM>>
>> expires: 2018-03-21 09:42:04 UTC
>> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>> eku: id-kp-OCSPSigning
>> --
>> status: CA_UNREACHABLE
>> ca-error: Internal error
>> stuck: no
>> key pair storage:
>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>> certificate:
>> type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>> Certificate DB'
>> CA: dogtag-ipa-ca-renew-agent
>> issuer: CN=Certificate Authority,O=<<REALM>>
>> subject: CN=IPA RA,O=<<REALM>>
>> expires: 2018-03-21 09:42:29 UTC
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-kp-clientAuth
>> --
>> status: CA_UNREACHABLE
>> ca-error: Error 7 connecting to
>> http://<<ipa2.fqdn>>:8080/ca/ee/ca/profileSubmit: Couldn't connect to server.
>> stuck: no
>> key pair storage:
>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB'
>> CA: dogtag-ipa-renew-agent
>> issuer: CN=Certificate Authority,O=<<REALM>>
>> subject: CN=<<ipa2.fqdn>>,O=<<REALM>>
>> expires: 2018-06-27 07:01:38 UTC
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku:
>> id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>> --
>>
>> Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2)
>> but shouldn't node1 be the renewal master? Restarting httpd, certmonger and
>> pki-tomcat don't seem to help, time traveling helped on other certs but not
>> on these.
>>
>> Directory service seems to work if I start it manually but
>> ipa-server-upgrade fails on directory server not starting with "No ports
>> specified" so something wrong with it or is it the certificates?
>> --
>> ipa-server-upgrade
>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>> [1/10]: stopping directory server
>> [2/10]: saving configuration
>> [3/10]: disabling listeners
>> [4/10]: enabling DS global lock
>> [5/10]: starting directory server
>>
>> --
>> <<ipa2.fqdn>> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] -
>> EMERG - main - Fatal Error---No ports specified. Exiting now.
>> --
>>
>> Also certmonger has issues:
>> --
>> dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last):
>>
>> File
>> "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 541, in
>> <module>
>>
>> sys.exit(main())
>>
>> File
>> "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 515, in main
>>
>> kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 43, in
>> kinit_keytab
>>
>> cred = gssapi.Credentials(name=name, store=store,
>> usage='initiate')
>>
>> File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",
>> line 64, in __new__
>>
>> store=store)
>>
>> File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",
>> line 148, in acquire
>>
>> usage)
>>
>> File "ext_cred_store.pyx", line 182, in
>> gssapi.raw.ext_cred_store.acquire_cred_from
>> (gssapi/raw/ext_cred_store.c:1732)
>>
>> GSSError: Major (851968): Unspecified GSS failure. Minor code
>> may provide more information, Minor (2529639068): Cannot contact any KDC for
>> realm '<<REALM>>'
>> --
>>
>> but KDCs should be able to be resolved even from ipa node2
>> --
>> nslookup -type=srv _kerberos._tcp.<<REALM>>
>> Server: <<ipa1.ip>>
>> Address: <<ipa1.ip>>#53
>>
>> _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa1.fqdn>>.
>> _kerberos._tcp.<<REALM>> service = 0 100 88 <<ipa2.fqdn>>.
>> --
>>
>> For testing purposes I turned off firewall on ipa node1
>>
>>
>> Eemeli
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- [email protected]
>> To unsubscribe send an email to
>> [email protected]
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines:
>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/[email protected]
>> o rahosted.org/message/XOUL2VQ26BKQHNY2XB3CDSJRKYQCHJ3X/
>>
>
> _______________________________________________
> FreeIPA-users mailing list -- [email protected]
> To unsubscribe send an email to
> [email protected]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[email protected]
> rahosted.org/message/XP7I6WTBMX2PYDBSI2OBCRGQA3HCRNWE/
>
_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]/message/HF2ORHTMRG6T6W7I62B4DZ4TD7FFNF54/
