On 8/10/20 7:56 PM, Scott Z. via FreeIPA-users wrote:
On the failing node, the output of "getcert list" does not show any expired certs.  I have hand-copied the info info this email below (it's interesting to note that while the other IdM servers are tracking 9 certs, the problem server is only tracking 8):

It looks like the server is missing the tracking request for 'Server-Cert cert-pki-ca'. You can add tracking for the cert with the nickname 'Server-Cert cert-pki-ca' in /etc/pki/pki-tomcat/alias with a command like this one:

getcert start-tracking -n 'Server-Cert cert-pki-ca' -d /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P <pin> and the PIN can be found in /etc/pki/pki-tomcat/password.conf in the line containing internal=<pin>

flo

Number of certificates and requests being tracked: 8
Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=<servername>,O=<domain>
subject: CN=<servername>,O=<domain>
expires: 2020-09-12 19:51:34 UTC
principal name: krbtgt/<domain>
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=CA Audit,O=<domain>
expires: 2021-08-10 17:20:21 UTC
key usage: digitialSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=OCSP Subsystem,O=<domain>
expires: 2021-08-10 17:19:42 UTC
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=CA Subsystem,O=<domain>
expires: 2021-08-10 17:19:51 UTC
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=Certificate Authority,O=<domain>
expires: 2037-09-28 14:29:02 UTC
key usage: digitialSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/var/lib/ipa/ra-agent.key'
certificate: type=NSSDB,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=IPA RA,,O=<domain>
expires: 2021-08-10 17:20:41 UTC
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<domain>/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<domain>
subject: CN=<server>,O=<domain>
expires: 2021-09-09 19:53:33 UTC
principal name: ldap/<serverFQDN@domain>
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <domain>
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<domain>
subject: CN=<server>,O=<domain>
expires: 2021-09-09 19:51:45 UTC
principal name: HTTP/<serverFQDN@domain>
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes

Thank you so much again!
Scot



------------------------------------------------------------------------
*From:* Florence Blanc-Renaud <[email protected]>
*Sent:* Thursday, August 6, 2020 2:46 AM
*To:* FreeIPA users list <[email protected]>
*Cc:* Scott Z. <[email protected]>
*Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
Thanks much for the assistance.  Here is where I am with your suggestions:
1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old (almost a year old actually, I assume IPA only checks it when it first starts up so it didn't care that it was expired until the server was rebooted?)

certmonger checks the certificate validity periodically (configurable in
certmonger.conf) and tries multiple times to renew soon-to-expire certs.
The system probably had an issue that was not detected and the cert
reached its expiration date.


2) ran ipactl start --ignore-service-failures
         a. most services started, obviously pki-tomcatd did 
not
3) ran "kinit admin"
         a. was forced to change the password, but otherwise 
nothing happened
4) Ran "ipa config-show |grep -i master
        a. I see that the IPA CA renewal master is a different 
idm machine.
5) Ran "getcert list | grep -E "Request|certificate:|expires:"
        a.I see all certs are currently valid (none expired)
6) Ran the command "getcert list" on the problem server, but I cannot paste the output here because it's on an airgaped environment so while I apologize for this and realize it makes things more difficult, perhaps if you tell me what I should be looking for or more specifically what you're interested in I can pluck that out and manually include it here? So in summary, it is indeed an expired "Server-Cert cert-pki-ca' certificate on the problem server, and it can theoretically be renew by the Master at this time.
The interesting part is the list of expired certs on the failing node
(is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
instructions are available here:
https://access.redhat.com/solutions/3357331 How do I manually renew
Identity Management (IPA) certificates on RHEL7 after they have expired?
(Replica IPA Server)

flo

Many thanks!
Scott

------------------------------------------------------------------------
*From:* Florence Blanc-Renaud <[email protected]>
*Sent:* Monday, August 3, 2020 9:34 PM
*To:* FreeIPA users list <[email protected]>
*Cc:* Scott Z. <[email protected]>
*Subject:* Re: [Freeipa-users] pki-tomcatd not starting
On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
Not sure I'm sending this to the right place, but here it goes.  I inherited a FreeIPA/Identity Manager setup in an enclave (no internet access) environment that is running into problems.  There are at least 3 different IdM servers running in the environment spread out across different geographical areas.  One of those areas suffered an unschedule power outage recently, and ever since we brought everything back up, the IdM server for this region is having an issue.  Please bear with me as I have zero formal experience, training, or real knowledge with IdM.

Logging in to the serverv (it's a VM server, running Centos 7.5), I run "ipactl status" and it shows "Directory Service: STOPPED".  I then run "ipactl restart", and things go fine until it gets to "Starting pki-tomcatd Service", where it hangs for quite some time before failing to start and killing all the other services.  I check the log at /var/log/pki/pki-tomcat/ca/debug and I see various errors such as (forgive any mistypings, I have to manually type these in as I can't import or screen capure the logs and put them in this message): "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid: Invalid certificate: (-8181) Peer's Certificate has expired/"
And slightly further down in the same log:
"/Cannot reset factory: connections not all returned/"
"/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset LDAP connection factory because some connections are still outstanding/"
... still further down"
"/returnConn:mNumConns now 3 Invalid class name repositorytop/"

Assuming I have some weird certificate issue with this server in particular, I try to run a few more commands: "certutil -L -d /etc/httpd/alias"  --> returns a Server-Cert listing with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C for it's attributes.  Comparing to a second IdM server in this environment, it seems to be missing a "Signing-Cert"?

Hi,
PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
has the nickname 'Server-Cert cert-pki-ca'. You should check that this
one is not expired with:
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
| grep 'Not '

If the certificate is indeed expired, it will have to be renewed but you
need first to find which IPA server is the CA renewal master. On your
server, force a service start and check the CA renewal master:
# ipactl start --ignore-service-failures
# kinit admin
# ipa config-show | grep "renewal master"
     IPA CA renewal master: server.domain.com

You need to make sure that all the certificates are valid on the CA
renewal master:
(on the CA renewal master)# getcert list | grep -E
"Request|certificate:|expires:"

- if the CA renewal master is not OK, please post the output of "#
getcert list" (without the grep) on the CA renewal master. This node
will have to be repaired first.
- if the CA renewal master is OK, please post the output of "# getcert
list" (also without the grep) on the failing node.

We'll be able to help based on this information.
flo

I also did a "getcert list", and all certs it has show that they expire in the future (nothing shows as bein currently expired).

I'm confused; it seems to that it is seeing an expired cert *somewhere*, but how do I track down which 'peer' the log file is talking about that has an expired cert?  Meanwhile none of the linux clients that point to this IdM server are allowing people to log in/authenticate.
Many thanks for any help!
Scott


_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]



_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]



_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]

_______________________________________________
FreeIPA-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedorahosted.org/archives/list/[email protected]

Reply via email to