[Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start

Tue, 26 Jun 2018 11:29:09 -0700 

On 06/26/2018 09:58 AM, Jokinen Eemeli via FreeIPA-users wrote:

Hello!

Thank you for your answers by the way, seems like we're getting closer and 
closer every step although haven't had a breakthrough yet... At least I feel 
like I understand the structure of IPA better alredy! A bit long message 
incoming... :)

First getcert list. Some sites say that there should be 9 certificates listed 
as of ipa-server 4.5

--
getcert list
Number of certificates and requests being tracked: 8.
Request ID '20160331084233':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=CA Audit,O=<<DOMAIN>>
         expires: 2018-03-21 09:42:06 UTC
         key usage: digitalSignature,nonRepudiation
         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-ca"
         track: yes
         auto-renew: yes
Request ID '20160331084234':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=OCSP Subsystem,O=<<DOMAIN>>
         expires: 2018-03-21 09:42:04 UTC
         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
         eku: id-kp-OCSPSigning
         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"ocspSigningCert cert-pki-ca"
         track: yes
         auto-renew: yes
Request ID '20160331084236':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=Certificate Authority,O=<<DOMAIN>>
         expires: 2036-03-31 08:42:02 UTC
         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
         pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
         post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert 
cert-pki-ca"
         track: yes
         auto-renew: yes
Request ID '20160331084238':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>>
         expires: 2020-02-11 09:58:22 UTC
         key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth
         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert 
"Server-Cert cert-pki-ca”
         track: yes
         auto-renew: yes
Request ID '20160331084308':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/dirsrv/slapd-<<REALM>>/pwdfile.txt'
         certificate: 
type=NSSDB,location='/etc/dirsrv/slapd-<<REALM>>',nickname='Server-Cert',token='NSS
 Certificate DB'
         CA: IPA
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>>
         expires: 2020-03-04 09:58:32 UTC
         principal name: ldap/<<ipa1.fqdn>>@<<DOMAIN>>
         key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth
         pre-save command:
         post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <<REALM>>
         track: yes
         auto-renew: yes
Request ID '20160331085008':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
         certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
         CA: IPA
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=<<ipa1.fqdn>>,O=<<DOMAIN>>
         expires: 2020-03-04 09:58:23 UTC
         principal name: HTTP/<<ipa1.fqdn>>@<<DOMAIN>>
         key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth
         pre-save command:
         post-save command: /usr/lib64/ipa/certmonger/restart_httpd
         track: yes
         auto-renew: yes
Request ID '20180611071929':
         status: MONITORING
         stuck: no
         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=IPA RA,O=<<DOMAIN>>
         expires: 2018-03-21 09:42:29 UTC
         key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth
         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
         track: yes
         auto-renew: yes
Request ID '20180615083528':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=Certificate Authority,O=<<DOMAIN>>
         subject: CN=CA Subsystem,O=<<DOMAIN>>
         expires: 2018-03-21 09:42:05 UTC
         key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
         eku: id-kp-serverAuth,id-kp-clientAuth
         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert 
cert-pki-ca"
         track: yes
         auto-renew: yes
--


Hi,
the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejected" (see [1] for the return codes). This probably happens because the cert for IPA RA is no longer valid (this cert is used to authenticate to Dogtag, and without proper authentication any renewal op is refused).
The expired certificates all expire on 2018-03-21. On the other hand, ServerCert cert-pki-ca, slapd and httpd certificates were properly renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Before" 
# certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before"
You need then to find a common date where all the certificates are valid (ie before 2018-03-21 so that the expired certs are not expired yet, and after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and look in the journal if the renewal goes smoothly or if there are errors that could point you in the right direction.
You can also find instructions on this blog post [2] to increase the log level for the renewal. 

HTH,
Flo

[1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46
[2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/ 


Next journalctl... I've tried changing the date of the server back to older 
days to get certmonger automatically renew them. Should I try this one again?

--
journalctl -u certmonger
-- Logs begin at Mon 2018-06-25 17:46:25 EEST, end at Tue 2018-06-26 10:43:30 
EEST. --
Jun 25 17:46:27 <<ipa1.fqdn>> certmonger[16802]: Certificate named "subsystemCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 25 17:46:29 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16804]: 
dogtag-ipa-renew-agent returned 2
Jun 25 17:46:36 <<ipa1.fqdn>> certmonger[16822]: Certificate named "auditSigningCert cert-pki-ca" 
in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 25 17:46:39 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16824]: 
dogtag-ipa-renew-agent returned 2
Jun 25 17:46:41 <<ipa1.fqdn>> certmonger[16839]: Certificate in file 
"/var/lib/ipa/ra-agent.pem" is no longer valid.
Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 25 17:46:43 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[16841]: 
dogtag-ipa-renew-agent returned 2
...
Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:40:47 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2530]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:40:48 <<ipa1.fqdn>> certmonger[2546]: Certificate named "ocspSigningCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:40:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2548]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:15 <<ipa1.fqdn>> certmonger[2580]: Certificate named "subsystemCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:17 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2582]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:18 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2594]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:20 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2608]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:21 <<ipa1.fqdn>> certmonger[2624]: Certificate named "ocspSigningCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:24 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2626]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:48 <<ipa1.fqdn>> certmonger[2667]: Certificate named "subsystemCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 26 10:41:50 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2669]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:51 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2682]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:53 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2697]: 
dogtag-ipa-renew-agent returned 2
Jun 26 10:41:54 <<ipa1.fqdn>> certmonger[2713]: Certificate named "ocspSigningCert cert-pki-ca" in 
token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" is no longer valid.
Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: 
Forwarding request to dogtag-ipa-renew-agent
Jun 26 10:41:57 <<ipa1.fqdn>> dogtag-ipa-ca-renew-agent-submit[2715]: 
dogtag-ipa-renew-agent returned 2
--

About versions:
OS CentOS 7.5.1804
Current IPA version 4.5.4-10.el7.centos.1 (from ipaupgrade.log)
Previous IPA version 4.2.0-15.0.1.el7.centos.6 (from ipaserver-install.log)
The date of the ipaserver-install.log is 2016.03.31 so exactly 720 days before 
the expire date of those 4 certificates...

I tought I had upgraded it once before but probably I just remember it wrong 
(we have a test environment also and it might be that I updated that one as 
part of troubleshooting process of another problem) because can't find any mark 
of it.


Eemeli

-----Original Message-----
From: Florence Blanc-Renaud [mailto:[email protected]]
Sent: tiistai 26. kesäkuuta 2018 10.27
To: FreeIPA users list <[email protected]>
Cc: Jokinen Eemeli <[email protected]>
Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade 
doesn't complete, pki-tomcatd won't start

On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote:

Hi!

The node 1 is the Renewal Master
--
ldapsearch -D cn=directory\ manager -W -LLL -b
cn=masters,cn=ipa,cn=etc,BASEDN '(&(cn=CA)(ipaConfigString=caRenewalMaster))' 
dn Enter LDAP Password:
dn: cn=CA,cn=<<ipa1.fqdn>>,cn=masters,cn=ipa,cn=etc,BASEDN
--


OK, so we know that your host node1 is the renewal master and it has 4 expired 
certificates. What is the full output of getcert list?

The journal will show why it was not able to renew them:
# journalctl -u certmonger

Can you also provide the version of FreeIPA you are using, and the one you had before the 
upgrade? (can be found in /var/log/ipaupgrade.log with the string "IPA version 
4.xx", this file keeps the whole upgrade history).

Flo


Eemeli

-----Original Message-----
From: Florence Blanc-Renaud [mailto:[email protected]]
Sent: maanantai 25. kesäkuuta 2018 12.53
To: FreeIPA users list <[email protected]>
Cc: Jokinen Eemeli <[email protected]>
Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade:
ipa-server-upgrade doesn't complete, pki-tomcatd won't start

On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote:

Hi!

gssproxy up and running

--
systemctl status gssproxy
● gssproxy.service - GSSAPI Proxy Daemon
      Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; 
vendor preset: disabled)
      Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 
days ago
     Process: 3807 ExecStart=/usr/sbin/gssproxy -D (code=exited,
status=0/SUCCESS)
--

Also seems like there's some default configuration of gssproxy, no ipa.conf 
(googling said that there should probably be also ipa.conf?).

--
ls /etc/gssproxy/
24-nfs-server.conf  99-nfs-client.conf  gssproxy.conf
--


Hi,
you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file should 
be created during ipa-server-upgrade, but after the step restarting pki-tomcat.

So let's go back to our initial goal: finding which master is the
renewal master. You can use a ldapsearch query to find out the renewal
master:
# ldapsearch -D cn=directory\ manager -W -LLL -b cn=masters,cn=ipa,cn=etc,$BASEDN 
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn Enter LDAP Password:
dn:
cn=CA,cn=myrenewalmaster.domain.com,cn=masters,cn=ipa,cn=etc,$BASEDN

(replace BASEDN with your own setting that can be found in
/etc/ipa/default.conf)

Flo

_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to
[email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
rahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/



_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/[email protected]/message/FHKV7F3U4HEA2STDG64L5LKEYXMJVVES/


_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/[email protected]/message/EHXLPL4PPU7B2ZAKFLHZHW72Y2HT3K4R/

Reply via email to