Re: [Freeipa-users] Certificate system unavailable [solved]
On 08/07/2014 05:27 PM, Lucas Yamanishi wrote: On 08/07/2014 04:48 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: On 08/07/2014 01:25 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. The subsystem certificate renewal should be independent of the web (and host) certificates. I'd focus on getting the CA back up, then we can see about getting a new web server certificate. Can you share the output of: getcert list You'll probably want to obfuscate the output as it contains the PIN to the private key database of the CA. rob Here you go. I've also included `certutil -L` outputs. The *auditSigningCert* I tried resubmitting with the time rolled back. The post-save command was also updated, because it wasn't done a year or two back when it replaced our old CRL-signer. `getcert list`: ``` Number of certificates and requests being tracked: 7. [ snip ] What version of IPA is this? Sorry. It's 3.0.0-37.el6 on Scientific Linux 6x. 389ds is 1.2.11.15-32.el6_5 and Dogtag is 9.0.3-32.el6. You need to modify a few more of these. Take a look at http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master Thanks. That was in my notes to do for the resubmits. The CS.cfg changes were made a long while back, before the guide. I think the ipa-pki-proxy.conf change was inherited with an upgrade. Those are awesome, BTW, the rpm automated upgrades! The renew_ra_cert script, too. When you roll back time are you restarting the pki-cad service? I think I did, but I can't recall. I will be sure to do it this weekend when I try again. rob Since you pointed out that the certificates and ipa commands should not be dependent on each other I discovered that the host ticket needed renewing. The version was out of sync. Running `kinit -kt /etc/krb5.keytab host/badca.example@example.com` fixed the ipa commands and I now get the expected SSL_ERROR_EXPIRED_CERT_ALERT code when doing a cert-request. Is there anything else I should look at? -- - *question everything*learn something*answer nothing* Lucas Yamanishi -- Systems Administrator, ADNET Systems, Inc. NASA Space and Earth Science Data Analysis (606.9) 7515 Mission Drive, Suite A100 Lanham, MD 20706 * 301-352-4646 * 0xD354B2CB To follow up, the second try went well and everything is again running smoothly. I followed the `getcert stop-tracking`, `getcert start-tracking` procedure described in the *HOWTO Promote a CA* page for all certificates but the *auditSigningCert* on which I previously ran `getcert resubmit`. I'm not sure if it was related to the resubmit or some other side-effect, but the *auditSigningCert* saved into a new index of the NSS DB leaving two identically named indexes, whereas the others saved into their existing indexes on top of their old certificates. I don't know what effect the separate indexes might have had, if any, but I was worried a process selecting it by name would select the wrong cert. It was easy enough to fix by exporting them both in ASCII format, deleting them both, then importing them as a single file. certutil -L -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -a /root/auditSigningCert.crt certutil -D -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' certutil -D -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' certutil -A -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -i /root/auditSigningCert.crt -t ',,P' Also, for some reason the RA certificate didn't properly publish. Manually running the post-save script was all it took to fix it. /usr/lib64/ipa/certmonger/renew_ra_cert -- - *question everything*learn something*answer nothing* Lucas Yamanishi -- Systems Administrator, ADNET
[Freeipa-users] Certificate system unavailable
Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. -- - *question everything*learn something*answer nothing* Lucas Yamanishi -- Systems Administrator, ADNET Systems, Inc. NASA Space and Earth Science Data Analysis (606.9) 7515 Mission Drive, Suite A100 Lanham, MD 20706 * 301-352-4646 * 0xD354B2CB -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Certificate system unavailable
Lucas Yamanishi wrote: Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. The subsystem certificate renewal should be independent of the web (and host) certificates. I'd focus on getting the CA back up, then we can see about getting a new web server certificate. Can you share the output of: getcert list You'll probably want to obfuscate the output as it contains the PIN to the private key database of the CA. rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Certificate system unavailable
On 08/07/2014 01:25 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. The subsystem certificate renewal should be independent of the web (and host) certificates. I'd focus on getting the CA back up, then we can see about getting a new web server certificate. Can you share the output of: getcert list You'll probably want to obfuscate the output as it contains the PIN to the private key database of the CA. rob Here you go. I've also included `certutil -L` outputs. The *auditSigningCert* I tried resubmitting with the time rolled back. The post-save command was also updated, because it wasn't done a year or two back when it replaced our old CRL-signer. `getcert list`: ``` Number of certificates and requests being tracked: 7. Request ID '20130321103859': status: CA_UNREACHABLE ca-error: Error 35 connecting to https://badca.example.com:9443/ca/agent/ca/profileReview: SSL connect error. stuck: yes key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Audit,O=EXAMPLE.COM expires: 2014-07-31 21:29:35 UTC pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert auditSigningCert cert-pki-ca track: yes auto-renew: yes Request ID '20130321103900': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin='' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=OCSP Subsystem,O=EXAMPLE.COM expires: 2014-07-31 21:29:33 UTC eku: id-kp-OCSPSigning pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad ocspSigningCert cert-pki-ca track: yes auto-renew: yes Request ID '20130321103901': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin='' certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=CA Subsystem,O=EXAMPLE.COM expires: 2014-07-31 21:29:34 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/restart_pkicad subsystemCert cert-pki-ca track: yes auto-renew: yes Request ID '20130321103902': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-retrieve-agent-submit issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2014-07-31 21:30:34 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command:
Re: [Freeipa-users] Certificate system unavailable
Lucas Yamanishi wrote: On 08/07/2014 01:25 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. The subsystem certificate renewal should be independent of the web (and host) certificates. I'd focus on getting the CA back up, then we can see about getting a new web server certificate. Can you share the output of: getcert list You'll probably want to obfuscate the output as it contains the PIN to the private key database of the CA. rob Here you go. I've also included `certutil -L` outputs. The *auditSigningCert* I tried resubmitting with the time rolled back. The post-save command was also updated, because it wasn't done a year or two back when it replaced our old CRL-signer. `getcert list`: ``` Number of certificates and requests being tracked: 7. [ snip ] What version of IPA is this? You need to modify a few more of these. Take a look at http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master When you roll back time are you restarting the pki-cad service? rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Certificate system unavailable
On 08/07/2014 04:48 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: On 08/07/2014 01:25 PM, Rob Crittenden wrote: Lucas Yamanishi wrote: Hello, I'm a bit of a pickle with the PKI system. I have three replicas, but only one contains the CA. I realize how poor a decision it was to do that. I plan to create more complete replicas, but right now I can't even create a replica file, much less a full replica. The problem started when the CA subsystem certificates expired. I read several threads explaining how to roll back time and renew them, but I then discovered that the host and HTTP certificates for the server were missing. I checked for backups, but we erroneously did not cover those files. Because they are missing I was unable to rewnew any certificates. Is there a way to manually create host and service certificates? When I search for this, the manual procedure listed in the documentation requires `ipa cert-request` which does not work. I did try installing a self-signed cert for HTTP with `ipa-server-certinstall`. That changed the errors, but the commands still fail. The pki-ca services is running OK, as far as I can tell. I also tried adding a CA instance to one of the other replicas with `ipa-ca-install`, but it failed during the configuration phase. The subsystem certificate renewal should be independent of the web (and host) certificates. I'd focus on getting the CA back up, then we can see about getting a new web server certificate. Can you share the output of: getcert list You'll probably want to obfuscate the output as it contains the PIN to the private key database of the CA. rob Here you go. I've also included `certutil -L` outputs. The *auditSigningCert* I tried resubmitting with the time rolled back. The post-save command was also updated, because it wasn't done a year or two back when it replaced our old CRL-signer. `getcert list`: ``` Number of certificates and requests being tracked: 7. [ snip ] What version of IPA is this? Sorry. It's 3.0.0-37.el6 on Scientific Linux 6x. 389ds is 1.2.11.15-32.el6_5 and Dogtag is 9.0.3-32.el6. You need to modify a few more of these. Take a look at http://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master Thanks. That was in my notes to do for the resubmits. The CS.cfg changes were made a long while back, before the guide. I think the ipa-pki-proxy.conf change was inherited with an upgrade. Those are awesome, BTW, the rpm automated upgrades! The renew_ra_cert script, too. When you roll back time are you restarting the pki-cad service? I think I did, but I can't recall. I will be sure to do it this weekend when I try again. rob Since you pointed out that the certificates and ipa commands should not be dependent on each other I discovered that the host ticket needed renewing. The version was out of sync. Running `kinit -kt /etc/krb5.keytab host/badca.example@example.com` fixed the ipa commands and I now get the expected SSL_ERROR_EXPIRED_CERT_ALERT code when doing a cert-request. Is there anything else I should look at? -- - *question everything*learn something*answer nothing* Lucas Yamanishi -- Systems Administrator, ADNET Systems, Inc. NASA Space and Earth Science Data Analysis (606.9) 7515 Mission Drive, Suite A100 Lanham, MD 20706 * 301-352-4646 * 0xD354B2CB -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
Re: [Freeipa-users] Certificate system unavailable
On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote: On Tue, February 18, 2014 20:45, Rob Crittenden wrote: Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates Yes, this did not work: curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Indeed - this works. Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML Yes, I can now see the login page HTML. If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This produces the same output for all 3 servers: certutil: certificate is valid This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. I ran # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt on ipa03, and restarted httpd. The ipa command is now *working* on ipa03. :) However it's *not working* a client who's /etc/ipa/default.conf points at ipa03. Do I have to refresh the PKI CA in /etc/pki/nssdb on every client? *ping* :) Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote: On Tue, February 18, 2014 20:45, Rob Crittenden wrote: Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates Yes, this did not work: curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Indeed - this works. Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML Yes, I can now see the login page HTML. If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This produces the same output for all 3 servers: certutil: certificate is valid This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. I ran # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt on ipa03, and restarted httpd. The ipa command is now *working* on ipa03. :) However it's *not working* a client who's /etc/ipa/default.conf points at ipa03. Do I have to refresh the PKI CA in /etc/pki/nssdb on every client? You shouldn't need to. Frankly I'm surprised this worked. What is the distro and what version of nss is installed? rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 20/02/14 21:19, Rob Crittenden wrote: Sigbjorn Lie wrote: On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote: On Tue, February 18, 2014 20:45, Rob Crittenden wrote: Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates Yes, this did not work: curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Indeed - this works. Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML Yes, I can now see the login page HTML. If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This produces the same output for all 3 servers: certutil: certificate is valid This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. I ran # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt on ipa03, and restarted httpd. The ipa command is now *working* on ipa03. :) However it's *not working* a client who's /etc/ipa/default.conf points at ipa03. Do I have to refresh the PKI CA in /etc/pki/nssdb on every client? You shouldn't need to. Frankly I'm surprised this worked. What is the distro and what version of nss is installed? rob This is RHEL 6.5 with nss-3.15.3-3.el6_5.x86_64. I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb before and after I updated it into text files, and diff'ed them. No differences was reported. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On 20/02/14 21:19, Rob Crittenden wrote: Sigbjorn Lie wrote: On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote: On Tue, February 18, 2014 20:45, Rob Crittenden wrote: Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates Yes, this did not work: curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Indeed - this works. Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML Yes, I can now see the login page HTML. If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This produces the same output for all 3 servers: certutil: certificate is valid This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. I ran # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt on ipa03, and restarted httpd. The ipa command is now *working* on ipa03. :) However it's *not working* a client who's /etc/ipa/default.conf points at ipa03. Do I have to refresh the PKI CA in /etc/pki/nssdb on every client? You shouldn't need to. Frankly I'm surprised this worked. What is the distro and what version of nss is installed? rob This is RHEL 6.5 with nss-3.15.3-3.el6_5.x86_64. I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb before and after I updated it into text files, and diff'ed them. No differences was reported. I can't think of a reason it would be using the sqlite database at all. You don't have NSS_DEFAULT_DB_TYPE set somewhere do you? I'd find it hard to believe that this would be set EVERYWHERE. If we want to brute force things, trying strace against a client that isn't working to confirm that it is trying to open cert9 might give us a data point at least. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 20/02/14 21:38, Rob Crittenden wrote: Sigbjorn Lie wrote: On 20/02/14 21:19, Rob Crittenden wrote: Sigbjorn Lie wrote: On Wed, February 19, 2014 13:45, Sigbjorn Lie wrote: On Tue, February 18, 2014 20:45, Rob Crittenden wrote: Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates Yes, this did not work: curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Indeed - this works. Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML Yes, I can now see the login page HTML. If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This produces the same output for all 3 servers: certutil: certificate is valid This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. I ran # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt on ipa03, and restarted httpd. The ipa command is now *working* on ipa03. :) However it's *not working* a client who's /etc/ipa/default.conf points at ipa03. Do I have to refresh the PKI CA in /etc/pki/nssdb on every client? You shouldn't need to. Frankly I'm surprised this worked. What is the distro and what version of nss is installed? rob This is RHEL 6.5 with nss-3.15.3-3.el6_5.x86_64. I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb before and after I updated it into text files, and diff'ed them. No differences was reported. I can't think of a reason it would be using the sqlite database at all. You don't have NSS_DEFAULT_DB_TYPE set somewhere do you? I'd find it hard to believe that this would be set EVERYWHERE. If we want to brute force things, trying strace against a client that isn't working to confirm that it is trying to open cert9 might give us a data point at least. I have NSS_DEFAULT_DB_TYPE set to sql. When I do a strace I can see that on /etc/pki/nssdb/cert9.db it does: access, stat, attempt open read-write which is denied, then open read only. Looks pretty normal? What is odd is the timestamp on the files in /etc/pki/nssdb/ on the different ipa servers ipa01 and ipa02, which is not working cert8.db is timestamped with the date we initially installed the ipa servers in 2012. cert9.db, key3.db, key4.db, secmod.db is timestamped in 2010, 2 years before the server was installed - so this must be the original file provided by the nss-sysinit package. ipa03, which is working after updating the nssdb: cert8.db and key3.db is timestamped with the date when I carried out the changes you proposed which this thread originally started with in january 2014 cert9.db and key4.db is timestamped with the date I ran the certutil command you recommended me to 2 days ago secmod.db is timestamped in 2010, 2 years before the server was installed. As I described earlier in this thread, I gave up on fixing the cert system on ipa03 and ended up removing it as a replica, creating a new replica file, and reinstalling using ipa-replica-install. This explains the date in january 2014. If cert9.db is the file being used, then the
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On 20/02/14 21:38, Rob Crittenden wrote: I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb before and after I updated it into text files, and diff'ed them. No differences was reported. I can't think of a reason it would be using the sqlite database at all. You don't have NSS_DEFAULT_DB_TYPE set somewhere do you? I'd find it hard to believe that this would be set EVERYWHERE. If we want to brute force things, trying strace against a client that isn't working to confirm that it is trying to open cert9 might give us a data point at least. I have NSS_DEFAULT_DB_TYPE set to sql. Oh, ok, that's why then. You're telling NSS to use sqlite databases and we only configure the older database style so the client isn't finding its CA cert. So you can either not set that or migrate all the client databases. I'm a little surprised the servers aren't blowing up on you too. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 20/02/14 23:08, Rob Crittenden wrote: Sigbjorn Lie wrote: On 20/02/14 21:38, Rob Crittenden wrote: I am surprised too. I dumped the PKI CA certificate from /etc/pki/nssdb before and after I updated it into text files, and diff'ed them. No differences was reported. I can't think of a reason it would be using the sqlite database at all. You don't have NSS_DEFAULT_DB_TYPE set somewhere do you? I'd find it hard to believe that this would be set EVERYWHERE. If we want to brute force things, trying strace against a client that isn't working to confirm that it is trying to open cert9 might give us a data point at least. I have NSS_DEFAULT_DB_TYPE set to sql. Oh, ok, that's why then. You're telling NSS to use sqlite databases and we only configure the older database style so the client isn't finding its CA cert. So you can either not set that or migrate all the client databases. I'm a little surprised the servers aren't blowing up on you too. Ohh so true...unset NSS_DEFAULT_DB_TYPE and it's all working fine! I can't believe it was something this silly! I've found the file where the NSS_DEFAULT_DB_TYPE is set to sql for our environment. This file has not been changed since Sep 2012. It's only set for a select amount of our accounts (mine being one of them) - that's why the servers isn't blowing up. And is why the webui is still working... We installed IPA in early 2012 and I've not had issues using the ipa command on any machines until a few weeks ago - and yes, NSS_DEFAULT_DB_TYPE=sql has been in the environment for my account the whole time. We recently installed a set of patches upgrading our servers to RHEL 6.5+(some updates) from 6.4. It would seem like something changed with this set of patches. And it also explains why this did not happen in the test environment as the same accounts are not being utilised there. Thank you for your assistance resolving these issues we've had recently. :) Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, February 17, 2014 17:59, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Mon, February 17, 2014 16:34, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has
now passed
for when the old certificate expired, and the ipa cli command no
longer works. The
webui is still working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been
marked as not
trusted by the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been
marked as not
trusted by the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been
marked as not
trusted by the user.) ipa: ERROR: cannot connect to Gettext('any of
the configured
servers', domain='ipa', localedir=None):
https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for
the output.
The issue happens when I'm logged on to any of the ipa servers, and if
I'm running the
ipa command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged
there.
I've sent you the full output in private. Could this be the reason for
why it fails?
ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False
Name: Certificate Key Usage
Critical: True
Usages:
Digital Signature
Non-Repudiation
Key Encipherment
Data Encipherment
No, that is normal for a server cert.
This pretty clearly looks like an SSL trust issue. Is this happening on
every client machine you run the ipa tool or just one?
You might want to compare the cert in /etc/pki/nssdb with the one on the
Apache server.
# certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA'
# certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Looks like the same certificate, just with different flags?
$ sudo certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' /tmp/ca1
$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA' /tmp/ca2
$ diff /tmp/ca1 /tmp/ca2
90a91,92
Valid CA
Trusted CA
A diff with context would confirm it but I'm assuming this is just the
code-signing trust which won't affect anything in this case. You'll notice
that some trust is
CT,C,C and some just CT,C,. The order is SSL,
S/MIME, code signing.
On what machine are you trying to use the ipa tool? Is it one of the
masters, all of them, enrolled clients?
It's the same error message when the ipa command is run directly on any of
the masters.
And it's the same error message if I run the ipa command on any of the
clients.
I do not have a working ipa command anywhere anymore.
Whatever is going on isn't likely related to the web server Apache
database as you get the same error out of each one. The client log you sent
confirmed that it tried
to contact each master. The SSL error we're getting is that the client
doesn't trust the CA that
signed the server certificate so this appears to be a problem on the client,
which begs the
question: all clients or just this one?
All clients.
NSS is smart enough to handle multiple certificates, it should pick the
newest one on startup.
Ok.
Where do you suggest I continue troubleshooting this issue?
Regards,
Siggi
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On what machine are you trying to use the ipa tool? Is it one of the masters, all of them, enrolled clients? It's the same error message when the ipa command is run directly on any of the masters. And it's the same error message if I run the ipa command on any of the clients. I do not have a working ipa command anywhere anymore. Ok, let's test out the cert that ipa is using. Try this on any one of the masters: $ curl https://`hostname`/ipa/xml Should fail with Peer certificate cannot be authenticated with known CA certificates $ curl --cacert /etc/ipa/ca.crt https://`hostname`/ipa/xml Should succeed in that you get the you are not logged in HTML page Ok, now unfortunately curl only handles the sql-style NSS databases so we can't fully reproduce it the same way that the IPA client is doing things, but here is an approximation: # certutil -A -d sql:/etc/pki/nssdb -n 'IPA CA' -t CT,C,C -a -i /etc/ipa/ca.crt $ curl https://`hostname`/ipa/xml You should see the login page HTML If you stick a -v in there it'll give you more verbose output which would be useful if any of these fail in an unexpected way. Whatever is going on isn't likely related to the web server Apache database as you get the same error out of each one. The client log you sent confirmed that it tried to contact each master. The SSL error we're getting is that the client doesn't trust the CA that signed the server certificate so this appears to be a problem on the client, which begs the question: all clients or just this one? All clients. NSS is smart enough to handle multiple certificates, it should pick the newest one on startup. Ok. Where do you suggest I continue troubleshooting this issue? We can also tackle this on the server side. Let's verify the server cert: # certutil -V -u V -n Server-Cert -d /etc/httpd/alias This is verified on server startup so I expect it to be valid, but doesn't hurt to try. Restarting the Apache process might be something to try as changes to the NSS database aren't picked up until a restart. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has now
passed for when
the old certificate expired, and the ipa cli command no longer works.
The webui is still
working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as
not trusted by
the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted by
the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted by
the user.) ipa: ERROR: cannot connect to Gettext('any of the configured
servers',
domain='ipa', localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa
command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged
there.
I've sent you the full output in private. Could this be the reason for why it
fails?
ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False
Name: Certificate Key Usage
Critical: True
Usages:
Digital Signature
Non-Repudiation
Key Encipherment
Data Encipherment
Regards,
Siggi
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, February 17, 2014 16:34, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has
now passed for
when the old certificate expired, and the ipa cli command no longer
works. The webui
is still working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cannot connect to Gettext('any of the
configured servers',
domain='ipa', localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa
command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged
there.
I've sent you the full output in private. Could this be the reason for why
it fails?
ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False
Name: Certificate Key Usage
Critical: True
Usages:
Digital Signature
Non-Repudiation
Key Encipherment
Data Encipherment
No, that is normal for a server cert.
This pretty clearly looks like an SSL trust issue. Is this happening on
every client machine you run the ipa tool or just one?
This is happening on every client I run the ipa tool.It also happens when I run
it directly on any
of the ipa servers.
You might want to compare the cert in /etc/pki/nssdb with the one on the
Apache server.
# certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA'
# certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The trust on each should be CT,C,C and can be checked with:
# certutil -L -d /etc/pki/nssdb
I see that I have two Server-Certs on ipa01. One of these expired recently, the
other is OK.
However I do not see any way to delete the old certificate only with certutil,
since they both
have the same name?
on ipa01:
$ sudo certutil -L -d /etc/pki/nssdb
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
IPA CA CT,C,C
$ sudo certutil -L -d /etc/httpd/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
EXAMPLE.COM IPA CA CT,C,C
Signing-Cert u,u,u
Server-Cert u,u,u
Server-Cert u,u,u
ipaCert u,u,u
on ipa02:
$ sudo certutil -L -d /etc/pki/nssdb
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
IPA CA CT,C,C
$ sudo certutil -L -d /etc/httpd/alias
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
EXAMPLE.COM IPA CA CT,C,
Server-Cert
Re: [Freeipa-users] Certificate system unavailable
On Mon, February 17, 2014 16:34, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has
now passed for
when the old certificate expired, and the ipa cli command no longer
works. The webui
is still working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cannot connect to Gettext('any of the
configured servers',
domain='ipa', localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa
command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged
there.
This error is recorded in the httpd error log. The same error is repeated on
all 3 ipa servers.
[Mon Feb 17 17:24:16 2014] [error] SSL Library Error: -12195 Peer does not
recognize and trust the
CA that issued your certificate
Regards,
Siggi
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, February 17, 2014 16:34, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has
now passed for
when the old certificate expired, and the ipa cli command no longer
works. The webui
is still working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked
as not trusted
by the user.) ipa: ERROR: cannot connect to Gettext('any of the
configured servers',
domain='ipa', localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa
command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged
there.
I've sent you the full output in private. Could this be the reason for why
it fails?
ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False
Name: Certificate Key Usage
Critical: True
Usages:
Digital Signature
Non-Repudiation
Key Encipherment
Data Encipherment
No, that is normal for a server cert.
This pretty clearly looks like an SSL trust issue. Is this happening on
every client machine you run the ipa tool or just one?
You might want to compare the cert in /etc/pki/nssdb with the one on the
Apache server.
# certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA'
# certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Looks like the same certificate, just with different flags?
$ sudo certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' /tmp/ca1
$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA' /tmp/ca2
$ diff /tmp/ca1 /tmp/ca2
90a91,92
Valid CA
Trusted CA
Regards,
Siggi
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote:
On Mon, February 17, 2014 16:34, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 17:18, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has now
passed for
when the old certificate expired, and the ipa cli command no longer works.
The webui
is still working just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted
by the user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted
by the user.) ipa: ERROR: cannot connect to Gettext('any of the configured
servers',
domain='ipa', localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa
command from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error logged there.
I've sent you the full output in private. Could this be the reason for why it
fails?
ipa: DEBUG: auth_certificate_callback: check_sig=True is_server=False
Name: Certificate Key Usage
Critical: True
Usages:
Digital Signature
Non-Repudiation
Key Encipherment
Data Encipherment
No, that is normal for a server cert.
This pretty clearly looks like an SSL trust issue. Is this happening on
every client machine you run the ipa tool or just one?
You might want to compare the cert in /etc/pki/nssdb with the one on the
Apache server.
# certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA'
# certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Looks like the same certificate, just with different flags?
$ sudo certutil -L -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' /tmp/ca1
$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA' /tmp/ca2
$ diff /tmp/ca1 /tmp/ca2
90a91,92
Valid CA
Trusted CA
A diff with context would confirm it but I'm assuming this is just the
code-signing trust which won't affect anything in this case. You'll
notice that some trust is CT,C,C and some just CT,C,. The order is SSL,
S/MIME, code signing.
On what machine are you trying to use the ipa tool? Is it one of the
masters, all of them, enrolled clients?
Whatever is going on isn't likely related to the web server Apache
database as you get the same error out of each one. The client log you
sent confirmed that it tried to contact each master. The SSL error we're
getting is that the client doesn't trust the CA that signed the server
certificate so this appears to be a problem on the client, which begs
the question: all clients or just this one?
NSS is smart enough to handle multiple certificates, it should pick the
newest one on startup.
rob
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Fri, January 31, 2014 20:32, Rob Crittenden wrote:
Sigbjorn Lie wrote:
On Fri, January 17, 2014 16:37, Rob Crittenden wrote:
Sigbjorn Lie wrote:
This worked better than expected. Thank you! :)
ipa01 and ipa02 seem to be happy again, getcert list no longer displays
any certificates
out of date, and all certificates in need of renewal within 28 days has
been renewed. The
webui also started working again and things seem to be back to normal.
ipa03 however is still having issues. I could not renew any certificates
on this server to
begin with, but I managed to renew the certificates for the directory
servers by changing
the xmlrpc url to another ipa server in /etc/ipa/default.conf and
resubmitting these
requests.
getcert resubmit -i request-id says SUBMITTING and the fails with
NEED_GUIDANCE after a short while for the certificates for the PKI service.
/var/log/messages says: certmonger: #033[?1034h28800 and python:
Updated certificate for ipaCert not available.
There is a lot of information in the /var/log/pki-ca/debug, but nothing
that I can easily distinguish as an error from all the other output.
Anything in particular
I
should look for?
Ok, so this is a bug in IPA related to python readline. Garbage is
getting inserted and causing bad things to happen,
https://fedorahosted.org/freeipa/ticket/4064
So the question is, are the certs available or not.
A number of the same certificates are shared amongst all the CAs. One
does the renewal and stuffs the result into
cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs
refer to that location for an updated cert and will load them if they are
updated.
Look to see if the certs are updated there. Given that you have 2
working masters I'm assuming that is the case, so it may just be a matter
of fixing the
python.
I could not get anywhere even after manually patching the python script as
mentioned in the
ticket you provided.
I ended up removing and re-adding the replica during a maintenance window.
For future
reference, what I did was to remove the replica as per the Identity
Management Guide on
docs.redhat.com. I then re-created the replica installation file and
installed the replica.
At this point Certmonger managed to retrieve new certificates for the
expired certificates, but
it kept segfaulting when it attempted to save the certificate to disk. I
restarted certmonger a
few times, but certmonger just ended up segfaulting over and over. I decided
to block the ipa
server off the network and change the date back to before the certs expired.
After the date was
changed I restarted certmonger. Certmonger managed to save the certs
successfully this time and
a getcert list now displays only certificates with an expire date of 2015
or 2016 and a status
of MONTORING.
I changed the date back to correct date and time and removed the iptables
rules. The replica
now works just fine.
Thank you for your assistance.
Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1032760
It would seem like we're still encountering some issues. The date has now
passed for when the old
certificate expired, and the ipa cli command no longer works. The webui is
still working just
fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cannot connect to Gettext('any of the configured servers',
domain='ipa',
localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
I've been looking through the old threads on the list for similar issues, but
these all seem to be
related to the date expiring and the issue is fixed by changing the date back
to before the
certificate expired and request a new certificate - which is what we've done as
well.
A getcert list shows valid certificates on all 3 ipa servers.
$ sudo getcert list|grep expires
expires: 2016-01-24 20:15:34 UTC
expires: 2015-12-28 14:25:19 UTC
expires: 2015-12-28 14:25:56 UTC
expires: 2015-12-28 14:25:56 UTC
expires: 2016-01-13 20:21:26 UTC
expires: 2016-01-24 20:15:32 UTC
expires: 2016-01-24 20:15:35 UTC
expires: 2015-12-28 14:25:56 UTC
Somehow I seem to have 2 Server-Cert on ipa01. But would that affect all the
servers?
$ sudo certutil -L -d /etc/httpd/alias
Certificate Nickname
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has now
passed for when the old
certificate expired, and the ipa cli command no longer works. The webui is
still working just
fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.)
ipa: ERROR: cannot connect to Gettext('any of the configured servers',
domain='ipa',
localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
rob
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has now
passed for when the
old certificate expired, and the ipa cli command no longer works. The
webui is still working
just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as
not trusted by the
user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as
not trusted by the
user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as
not trusted by the
user.) ipa: ERROR: cannot connect to Gettext('any of the configured
servers', domain='ipa',
localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa command
from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Regards,
Siggi
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote:
On Fri, February 14, 2014 15:29, Rob Crittenden wrote:
Sigbjorn Lie wrote:
It would seem like we're still encountering some issues. The date has now
passed for when the
old certificate expired, and the ipa cli command no longer works. The webui
is still working
just fine.
These are the errors I receive.
$ ipa user-find
ipa: ERROR: cert validation failed for
CN=serveripa03.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.) ipa: ERROR: cert validation failed for
CN=serveripa01.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.) ipa: ERROR: cert validation failed for
CN=serveripa02.example.com,O=EXAMPLE.COM
((SEC_ERROR_UNTRUSTED_ISSUER) Peer's certificate issuer has been marked as not
trusted by the
user.) ipa: ERROR: cannot connect to Gettext('any of the configured servers',
domain='ipa',
localedir=None): https://serveripa03.example.com/ipa/xml,
https://serveripa01.example.com/ipa/xml,
https://serveripa02.example.com/ipa/xml
This seems more like a client-side issue. Can you confirm that
/etc/ipa/ca.crt is correct and that the NSS database in /etc/pki/nssdb
contains the CA?
certutil -L -d /etc/pki/nssdb -n 'IPA CA'
The CA seem to be available. I ran the command on ipa01. See below for the
output.
The issue happens when I'm logged on to any of the ipa servers, and if I'm
running the ipa command
from a remote machine.
]$ sudo certutil -L -d /etc/pki/nssdb -n 'IPA CA'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Validity:
Not Before: Thu Jan 19 19:44:21 2012
Not After : Sun Jan 19 19:44:21 2020
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Subject Public Key Info:
Public Key Algorithm: PKCS #1 RSA Encryption
RSA Public Key:
Modulus:
Perhaps we can brute force things to see what is going on. We can pass
some extra options to the ipa tool to get ultra verbose output:
$ ipa -vv -e debug=True user-show admin
The thing to do is to check the server that it is communicating with and
check /var/log/httpd/errors to see if there is an equivalent error
logged there.
rob
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 01/31/2014 08:32 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 17, 2014 16:37, Rob Crittenden wrote: Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. I could not get anywhere even after manually patching the python script as mentioned in the ticket you provided. I ended up removing and re-adding the replica during a maintenance window. For future reference, what I did was to remove the replica as per the Identity Management Guide on docs.redhat.com. I then re-created the replica installation file and installed the replica. At this point Certmonger managed to retrieve new certificates for the expired certificates, but it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa server off the network and change the date back to before the certs expired. After the date was changed I restarted certmonger. Certmonger managed to save the certs successfully this time and a getcert list now displays only certificates with an expire date of 2015 or 2016 and a status of MONTORING. I changed the date back to correct date and time and removed the iptables rules. The replica now works just fine. Thank you for your assistance. Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1032760 rob This one might be related as well: https://bugzilla.redhat.com/show_bug.cgi?id=1040009. Martin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Fri, January 17, 2014 16:37, Rob Crittenden wrote: Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. I could not get anywhere even after manually patching the python script as mentioned in the ticket you provided. I ended up removing and re-adding the replica during a maintenance window. For future reference, what I did was to remove the replica as per the Identity Management Guide on docs.redhat.com. I then re-created the replica installation file and installed the replica. At this point Certmonger managed to retrieve new certificates for the expired certificates, but it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa server off the network and change the date back to before the certs expired. After the date was changed I restarted certmonger. Certmonger managed to save the certs successfully this time and a getcert list now displays only certificates with an expire date of 2015 or 2016 and a status of MONTORING. I changed the date back to correct date and time and removed the iptables rules. The replica now works just fine. Thank you for your assistance. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 01/31/2014 10:00 AM, Sigbjorn Lie wrote: On Fri, January 17, 2014 16:37, Rob Crittenden wrote: Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. I could not get anywhere even after manually patching the python script as mentioned in the ticket you provided. I ended up removing and re-adding the replica during a maintenance window. For future reference, what I did was to remove the replica as per the Identity Management Guide on docs.redhat.com. I then re-created the replica installation file and installed the replica. At this point Certmonger managed to retrieve new certificates for the expired certificates, but it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa server off the network and change the date back to before the certs expired. After the date was changed I restarted certmonger. Certmonger managed to save the certs successfully this time and a getcert list now displays only certificates with an expire date of 2015 or 2016 and a status of MONTORING. I changed the date back to correct date and time and removed the iptables rules. The replica now works just fine. Thank you for your assistance. Can you give us some core dumps from certmonger to see why it is crashing. We would like to fix crash bugs if we them. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sure thing! I'll send them to you in private. Regards Siggi Dmitri Pal d...@redhat.com wrote: On 01/31/2014 10:00 AM, Sigbjorn Lie wrote: On Fri, January 17, 2014 16:37, Rob Crittenden wrote: Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. I could not get anywhere even after manually patching the python script as mentioned in the ticket you provided. I ended up removing and re-adding the replica during a maintenance window. For future reference, what I did was to remove the replica as per the Identity Management Guide on docs.redhat.com. I then re-created the replica installation file and installed the replica. At this point Certmonger managed to retrieve new certificates for the expired certificates, but it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa server off the network and change the date back to before the certs expired. After the date was changed I restarted certmonger. Certmonger managed to save the certs successfully this time and a getcert list now displays only certificates with an expire date of 2015 or 2016 and a status of MONTORING. I changed the date back to correct date and time and removed the iptables rules. The replica now works just fine. Thank you for your assistance. Can you give us some core dumps from certmonger to see why it is crashing. We would like to fix crash bugs if we them. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On Fri, January 17, 2014 16:37, Rob Crittenden wrote: Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. I could not get anywhere even after manually patching the python script as mentioned in the ticket you provided. I ended up removing and re-adding the replica during a maintenance window. For future reference, what I did was to remove the replica as per the Identity Management Guide on docs.redhat.com. I then re-created the replica installation file and installed the replica. At this point Certmonger managed to retrieve new certificates for the expired certificates, but it kept segfaulting when it attempted to save the certificate to disk. I restarted certmonger a few times, but certmonger just ended up segfaulting over and over. I decided to block the ipa server off the network and change the date back to before the certs expired. After the date was changed I restarted certmonger. Certmonger managed to save the certs successfully this time and a getcert list now displays only certificates with an expire date of 2015 or 2016 and a status of MONTORING. I changed the date back to correct date and time and removed the iptables rules. The replica now works just fine. Thank you for your assistance. Sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1032760 rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be back to normal. ipa03 however is still having issues. I could not renew any certificates on this server to begin with, but I managed to renew the certificates for the directory servers by changing the xmlrpc url to another ipa server in /etc/ipa/default.conf and resubmitting these requests. getcert resubmit -i request-id says SUBMITTING and the fails with NEED_GUIDANCE after a short while for the certificates for the PKI service. /var/log/messages says: certmonger: #033[?1034h28800 and python: Updated certificate for ipaCert not available. There is a lot of information in the /var/log/pki-ca/debug, but nothing that I can easily distinguish as an error from all the other output. Anything in particular I should look for? Ok, so this is a bug in IPA related to python readline. Garbage is getting inserted and causing bad things to happen, https://fedorahosted.org/freeipa/ticket/4064 So the question is, are the certs available or not. A number of the same certificates are shared amongst all the CAs. One does the renewal and stuffs the result into cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX. The other CAs refer to that location for an updated cert and will load them if they are updated. Look to see if the certs are updated there. Given that you have 2 working masters I'm assuming that is the case, so it may just be a matter of fixing the python. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Certificate system unavailable
Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. What is the status on the failed certmonger requests? rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. Looks like this certificate is expired. This is the same output on all 3 of the ipa servers. How can this be fixed? # certutil -L -d /var/lib/pki-ca/alias/ -n auditSigningCert cert-pki-ca Certificate: Data: Version: 3 (0x2) Serial Number: 5 (0x5) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=DNS.DOMAIN Validity: Not Before: Thu Jan 19 19:44:24 2012 Not After : Wed Jan 08 19:44:24 2014 ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. Looks like this certificate is expired. This is the same output on all 3 of the ipa servers. How can this be fixed? # certutil -L -d /var/lib/pki-ca/alias/ -n auditSigningCert cert-pki-ca Certificate: Data: Version: 3 (0x2) Serial Number: 5 (0x5) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=DNS.DOMAIN Validity: Not Before: Thu Jan 19 19:44:24 2012 Not After : Wed Jan 08 19:44:24 2014 Go back in time to the 7th or 8th and run: # getcert resubmit -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca There may be other certs in a similar situation. getcert list will show you. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, January 13, 2014 16:34, Rob Crittenden wrote: Sigbjorn Lie wrote: On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. Looks like this certificate is expired. This is the same output on all 3 of the ipa servers. How can this be fixed? # certutil -L -d /var/lib/pki-ca/alias/ -n auditSigningCert cert-pki-ca Certificate: Data: Version: 3 (0x2) Serial Number: 5 (0x5) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=DNS.DOMAIN Validity: Not Before: Thu Jan 19 19:44:24 2012 Not After : Wed Jan 08 19:44:24 2014 Go back in time to the 7th or 8th and run: # getcert resubmit -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca There may be other certs in a similar situation. getcert list will show you. Ouch. That would be rather disruptive I suppose. There is quite a lot of activity going to this server, not to mention it's the primary ntp and dns server for the network. Do you suppose this todo list will work ? Firewall off the rest of the network, leaving the ipa server alone Stop ntpd Set date to 8th of January Run the getcert resubmit command. Change date back to correct date Start ntpd Remove the firewall rules How many of the services is required to be restarted for the renewal to work after the date is changed to the 7th? Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Hi, Thank you for your prompt reply Rob. On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu All the 3 ipa servers return u,u,Pu for auditSigningCert # certutil -L -d /var/lib/pki-ca/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI caSigningCert cert-pki-caCTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-cau,u,u If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. I have restarted the dirsrv for PKI-IPA, and the pki-cad service on all 3 IPA servers. What is the status on the failed certmonger requests? After I restarted dirsrv, pki-cad and then the httpd on ipa01 the status of the request is now: Request ID '20120119194518': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ipa01.dns.domain:443/ca/agent/ca/displayBySerial': [Errno -12269] (SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DNS-DOMAIN//pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DNS-DOMAIN subject: CN=ipa01.dns.domain,O=DNS-DOMAIN expires: 2014-01-19 19:45:18 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes However I cannot find the certificate that's expired? Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On Mon, January 13, 2014 16:17, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, Thank you for your prompt reply Rob. On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu All the 3 ipa servers return u,u,Pu for auditSigningCert # certutil -L -d /var/lib/pki-ca/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI caSigningCert cert-pki-caCTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. I have restarted the dirsrv for PKI-IPA, and the pki-cad service on all 3 IPA servers. What is the status on the failed certmonger requests? After I restarted dirsrv, pki-cad and then the httpd on ipa01 the status of the request is now: Request ID '20120119194518': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ipa01.dns.domain:443/ca/agent/ca/displayBySerial': [Errno -12269] (SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DNS-DOMAIN//pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DNS-DOMAIN subject: CN=ipa01.dns.domain,O=DNS-DOMAIN expires: 2014-01-19 19:45:18 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes However I cannot find the certificate that's expired? Can provide the output of getcert rather than ipa-getcert? It will show additional certificates that are issued/renewed outside of the IPA API. rob Sure, I'll send you the output in private so I don't have to remove the domain names. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com
Re: [Freeipa-users] Certificate system unavailable
On Mon, Jan 13, 2014 at 04:07:16PM +0100, Sigbjorn Lie wrote: After I restarted dirsrv, pki-cad and then the httpd on ipa01 the status of the request is now: Request ID '20120119194518': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ipa01.dns.domain:443/ca/agent/ca/displayBySerial': [Errno -12269] (SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DNS-DOMAIN//pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DNS-DOMAIN subject: CN=ipa01.dns.domain,O=DNS-DOMAIN expires: 2014-01-19 19:45:18 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes However I cannot find the certificate that's expired? That error message was the one the IPA server received and then relayed back to certmonger, so I'd expect that the expired certificate is the agent certificate that IPA uses when connecting to the CA's agent interface. That's stored in the NSS database in /etc/httpd/alias, with nickname ipaCert. HTH, Nalin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 13/01/14 19:13, Nalin Dahyabhai wrote: On Mon, Jan 13, 2014 at 04:07:16PM +0100, Sigbjorn Lie wrote: After I restarted dirsrv, pki-cad and then the httpd on ipa01 the status of the request is now: Request ID '20120119194518': status: CA_UNREACHABLE ca-error: Server failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ipa01.dns.domain:443/ca/agent/ca/displayBySerial': [Errno -12269] (SSL_ERROR_EXPIRED_CERT_ALERT) SSL peer rejected your certificate as expired.). stuck: yes key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DNS-DOMAIN//pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DNS-DOMAIN',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DNS-DOMAIN subject: CN=ipa01.dns.domain,O=DNS-DOMAIN expires: 2014-01-19 19:45:18 UTC eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes However I cannot find the certificate that's expired? That error message was the one the IPA server received and then relayed back to certmonger, so I'd expect that the expired certificate is the agent certificate that IPA uses when connecting to the CA's agent interface. That's stored in the NSS database in /etc/httpd/alias, with nickname ipaCert. Yes, the ipaCert certificate in /etc/httpd/alias/ is expired. Actually all certificates in /var/lib/pki-ca/alias/ is expired too, they all expired at the same date, within minutes of each other. It looks like they are the original certificates issued when I installed IPA, when I look at the Not Before timestamp of the certificates. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
Sigbjorn Lie wrote: On Mon, January 13, 2014 16:34, Rob Crittenden wrote: Sigbjorn Lie wrote: On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. Looks like this certificate is expired. This is the same output on all 3 of the ipa servers. How can this be fixed? # certutil -L -d /var/lib/pki-ca/alias/ -n auditSigningCert cert-pki-ca Certificate: Data: Version: 3 (0x2) Serial Number: 5 (0x5) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=DNS.DOMAIN Validity: Not Before: Thu Jan 19 19:44:24 2012 Not After : Wed Jan 08 19:44:24 2014 Go back in time to the 7th or 8th and run: # getcert resubmit -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca There may be other certs in a similar situation. getcert list will show you. Ouch. That would be rather disruptive I suppose. There is quite a lot of activity going to this server, not to mention it's the primary ntp and dns server for the network. Do you suppose this todo list will work ? Firewall off the rest of the network, leaving the ipa server alone Stop ntpd Set date to 8th of January Run the getcert resubmit command. Change date back to correct date Start ntpd Remove the firewall rules Looks good. I'd restart the certmonger service rather than resubmitting each individually. Be prepared for renewal to not succeed. For some reason it didn't on and before expiration time so whatever problem existed then likely still remains. So the question to ask is what will I do if renewal fails again? Nothing catastrophic will happen, but it will likely mean having to roll forward again, debug, roll back, try again, and perhaps more than once. It's hard to say w/o knowing why it failed in the first place. How many of the services is required to be restarted for the renewal to work after the date is changed to the 7th? The renewal itself should restart the required services. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Certificate system unavailable
On 13/01/14 19:37, Rob Crittenden wrote: Sigbjorn Lie wrote: On Mon, January 13, 2014 16:34, Rob Crittenden wrote: Sigbjorn Lie wrote: On Mon, January 13, 2014 15:58, Rob Crittenden wrote: Sigbjorn Lie wrote: Hi, I seem to have issues with the certificate system on my IPA installation. Looking up hosts in the IPA WEBUI on any of the IPA servers says Certificate format error: [Errno -8015] error (-8015) unknown. I also notice that hosts says the certificate system is unavailable. certmonger: Server failed request, will retry: 4301 (RPC failed at server. Certificate operation cannot be completed: Failure decoding Certificate Signing Request). Looking at the pki-ca logs on the ipa servers I see that some selftest failed: # tail -100 selftests.log 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Initializing self test plugins: 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instances 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 28697.main - [13/Jan/2014:15:06:33 CET] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] CAPresence: CA is present 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SystemCertsVerification: system certs verification failure 28697.main - [13/Jan/2014:15:06:34 CET] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! the pki-cad service is running and pki-cad status displays the ports available. /etc/init.d/pki-cad status pki-ca (pid 28697) is running... [ OK ] My main consern is that the certmonger requests for renew of certificates for LDAP on 2 out of 3 of the IPA servers has failed, and the current certificate is expiring the 19th of January, under a week from now. Do you have any suggestions to where I can start troubleshootng this issue? Check the trust on the audit certificate: # certutil -L -d /var/lib/pki-ca/alias/ ... auditSigningCert cert-pki-ca u,u,Pu If the trust is not u,u,Pu then you can fix it with: # certutil -M -d /var/lib/pki-ca/alias -n 'auditSigningCert cert-pki-ca' -t u,u,Pu Then restart the CA and it should be ok. Looks like this certificate is expired. This is the same output on all 3 of the ipa servers. How can this be fixed? # certutil -L -d /var/lib/pki-ca/alias/ -n auditSigningCert cert-pki-ca Certificate: Data: Version: 3 (0x2) Serial Number: 5 (0x5) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: CN=Certificate Authority,O=DNS.DOMAIN Validity: Not Before: Thu Jan 19 19:44:24 2012 Not After : Wed Jan 08 19:44:24 2014 Go back in time to the 7th or 8th and run: # getcert resubmit -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca There may be other certs in a similar situation. getcert list will show you. Ouch. That would be rather disruptive I suppose. There is quite a lot of activity going to this server, not to mention it's the primary ntp and dns server for the network. Do you suppose this todo list will work ? Firewall off the rest of the network, leaving the ipa server alone Stop ntpd Set date to 8th of January Run the getcert resubmit command. Change date back to correct date Start ntpd Remove the firewall rules Looks good. I'd restart the certmonger service rather than resubmitting each individually. Be prepared for renewal to not succeed. For some reason it didn't on and before expiration time so whatever problem existed then likely still remains. So the question to ask is what will I do if renewal fails again? Nothing catastrophic will happen, but it will likely mean having to roll forward again, debug, roll back, try again, and perhaps more than once. It's hard to say w/o knowing why it failed in the first place. How many of the services is required to be restarted for the renewal to work after the date is changed to the 7th? The renewal itself should restart the required services. This worked better than expected. Thank you! :) ipa01 and ipa02 seem to be happy again, getcert list no longer displays any certificates out of date, and all certificates in need of renewal within 28 days has been renewed. The webui also started working again and things seem to be
