Each of the CA subsystem certificates would trigger a restart during
renewal. This generally caused one or more of the renewals to fail due
to the CA being down.
We also need to fix the trust on the audit cert post-installation. It
was possible that both certmonger and certutil could have the NSS
database open read/write which is almost guaranteed to result in corruption.
So intead I picked the audit cert as the "lead" cert. It will handle
restarting the CA.
It will also wait until all the other CA subsystem certs are in a
MONITORING state before trying to update the trust. This should prevent
the multiple read/write problem.
The CA wasn't actually working post-renewal anyway because the user it
uses to bind to DS wasn't being updated properly. certmap.conf is
confiugred to compare the cert provided by the client with that stored
in LDAP and since we weren't updating it, dogtag couldn't properly bind
to its own DS instance.
We also update a ou=People entry for the RA agent cert so I pulled that
updating code into cainstance.py for easier sharing.
Finally, the wrong service name was being used for tomcat to do the
restart. This is fixed. I've tested this with 3.1/dogtag 10 but it
should work with dogtag 9 as well (which uses a different service naming
convention).
This is how I test:
- ipa-server-install ...
- getcert list | grep expires
- examine the first four certs, pick an expiration date ~28 days prior
- date MMDDhhmmCCYY
- getcert list|grep status
Wait until all but one is in MONITORING. That last one should be the
audit cert.
I usually at this point switch to watching a tail of /var/log/messages
until the CA restarts.
Confirm that things are working with:
- ipa cert-show 1
To really be sure, use the ipa cert-request command to issue a new cert.
Ideally you'll verify that things are working, then trigger another
renewal event. Do the getcert list|grep expires to renew the HTTP/DS
server certs, then do this again for the CA subsystem certs.
It should come up again.
rob
>From e7037ba78f6907611e8342f6d3737b6d92534c99 Mon Sep 17 00:00:00 2001
From: Rob Crittenden <rcrit...@redhat.com>
Date: Tue, 2 Dec 2014 13:18:36 -0500
Subject: [PATCH] Limit tomcat restarts during renewal to prevent NSS db
corruption.
This also reduces the number of restarts from 4 to 1 making the process
much faster.
Fix a number of bugs in renewal. The CA wasn't actually being restarted
at all due to a naming change upstream. In python we need to reference
services using python-ish names but the service is pki-cad. We need a
translation for non-Fedora systems as well.
Update the CA ou=People entry when he CA subsystem certificate is
renewed. This certificate is used as an identity certificate to bind
to the DS instance.
https://fedorahosted.org/freeipa/ticket/3292
---
install/restart_scripts/renew_ca_cert | 81 ++++++++++++++++++++++++-----------
install/restart_scripts/renew_ra_cert | 54 +++--------------------
ipapython/certmonger.py | 18 ++++----
ipaserver/install/cainstance.py | 70 +++++++++++++++++++++++++++++-
4 files changed, 140 insertions(+), 83 deletions(-)
diff --git a/install/restart_scripts/renew_ca_cert b/install/restart_scripts/renew_ca_cert
index 5317835fc6ad7b598290fa5387be0afa46a6ca5b..361e56f9a5987ec664b4570adc182676bf51f03e 100644
--- a/install/restart_scripts/renew_ca_cert
+++ b/install/restart_scripts/renew_ca_cert
@@ -34,8 +34,10 @@ from ipapython import services as ipaservices
from ipapython import ipautil
from ipapython import dogtag
from ipaserver.install import certs
+from ipaserver.install.cainstance import update_people_entry
from ipaserver.plugins.ldap2 import ldap2
from ipaserver.install.cainstance import update_cert_config
+from ipapython import certmonger
# This script a post-cert-install command for certmonger. When certmonger
# has renewed a CA subsystem certificate a copy is put into the replicated
@@ -82,34 +84,63 @@ except Exception, e:
finally:
shutil.rmtree(tmpdir)
-# Fix permissions on the audit cert if we're updating it
-if nickname == 'auditSigningCert cert-pki-ca':
- db = certs.CertDB(api.env.realm, nssdir=alias_dir)
- args = ['-M',
- '-n', nickname,
- '-t', 'u,u,Pu',
- ]
- try:
- db.run_certutil(args)
- except ipautil.CalledProcessError:
- syslog.syslog(syslog.LOG_ERR, 'Updating trust on certificate %s failed in %s' % (nickname, db.secdir))
-
update_cert_config(nickname, cert)
-syslog.syslog(
- syslog.LOG_NOTICE, 'certmonger restarted %sd instance %s to renew %s' %
- (dogtag_instance, dogtag_instance, nickname))
-
-# We monitor 3 certs that are all likely to be renewed by certmonger more or
-# less at the same time. Each cert renewal is going to need to restart
-# the CA. Add a bit of randomness in this so not all three try to start it
-# at the same time. A restart is needed for each because there is no guarantee
-# that they will all be renewed at the same time.
-pause = random.randint(10,360)
-syslog.syslog(syslog.LOG_NOTICE, 'Pausing %d seconds to restart pki-ca' % pause)
-time.sleep(pause)
+if nickname == 'subsystemCert cert-pki-ca':
+ update_people_entry('pkidbuser', cert)
+
+# We pick one certificate, the audit certificate, to trigger the completion
+# of subsystem renewal. This achieves two things:
+# 1. We synchronize access to the NSS database so multiple scripts aren't
+# accessing it at once.
+# 2. We reduce the number of restarts of the server to one.
+if nickname != 'auditSigningCert cert-pki-ca':
+ sys.exit(0)
+
+# Find all the dogtag certificates and wait until their are all renewed.
+# The audit cert is the exception, since it's this one, and should be in
+# POST_SAVED_CERT.
+
+ids = certmonger.get_requests_for_dir(configured_constants.ALIAS_DIR)
+monitored = 0
+iterations = 0
+max_iterations = 100 # 10 minutes
+
+while monitored != len(ids) and iterations < max_iterations:
+ monitored = 0
+ time.sleep(6)
+ for id in ids:
+ state = certmonger.get_request_value(id, 'state').strip()
+ if state == 'MONITORING':
+ monitored += 1
+ elif state == 'POST_SAVED_CERT':
+ nickname = certmonger.get_request_value(id, 'cert_nickname').strip()
+ if nickname == 'auditSigningCert cert-pki-ca':
+ monitored += 1
+ iterations += 1
+
+if monitored != len(ids):
+ syslog.syslog(syslog.LOG_ERR, '%s: Giving up after %d seconds' % (sys.argv[0], max_iterations * 6))
+ sys.exit(1)
+
+# Fix trust on the audit cert
+db = certs.CertDB(api.env.realm, nssdir=alias_dir)
+args = ['-M',
+ '-n', nickname,
+ '-t', 'u,u,Pu',
+ ]
+try:
+ db.run_certutil(args)
+ syslog.syslog(syslog.LOG_NOTICE, 'Updated trust on certificate %s in %s' % (nickname, db.secdir))
+except ipautil.CalledProcessError:
+ syslog.syslog(syslog.LOG_ERR, 'Updating trust on certificate %s failed in %s' % (nickname, db.secdir))
+
+syslog.syslog(syslog.LOG_NOTICE, 'Restarting %sd' % dogtag_instance)
try:
- ipaservices.knownservices.pki_cad.restart(dogtag_instance)
+ if configured_constants.DOGTAG_VERSION == 9:
+ ipaservices.knownservices.pki_cad.restart(dogtag_instance)
+ else:
+ ipaservices.knownservices.pki_tomcatd.restart(dogtag_instance)
except Exception, e:
syslog.syslog(syslog.LOG_ERR, "Cannot restart %sd: %s" %
(dogtag_instance, str(e)))
diff --git a/install/restart_scripts/renew_ra_cert b/install/restart_scripts/renew_ra_cert
index 1f359062b49eed400aaa7b6aeea4742253707b00..a70ba5c1a842f5724ea655d579f5a71f69d0ea21 100644
--- a/install/restart_scripts/renew_ra_cert
+++ b/install/restart_scripts/renew_ra_cert
@@ -25,13 +25,11 @@ import tempfile
import syslog
import time
from ipapython import services as ipaservices
-from ipapython.certmonger import get_pin
from ipapython import ipautil
from ipaserver.install import certs
-from ipaserver.install.cainstance import DEFAULT_DSPORT
+from ipaserver.install.cainstance import update_people_entry
from ipalib import api
from ipapython.dn import DN
-from ipalib import x509
from ipalib import errors
from ipaserver.plugins.ldap2 import ldap2
import ldap as _ldap
@@ -41,52 +39,10 @@ api.finalize()
# Fetch the new certificate
db = certs.CertDB(api.env.realm)
-cert = db.get_cert_from_db('ipaCert', pem=False)
-serial_number = x509.get_serial_number(cert, datatype=x509.DER)
-subject = x509.get_subject(cert, datatype=x509.DER)
-issuer = x509.get_issuer(cert, datatype=x509.DER)
+dercert = db.get_cert_from_db('ipaCert', pem=False)
# Load it into dogtag
-dn = DN(('uid','ipara'),('ou','People'),('o','ipaca'))
-
-try:
- dm_password = get_pin('internaldb')
-except IOError, e:
- syslog.syslog(syslog.LOG_ERR, 'Unable to determine PIN for CA instance: %s' % e)
- sys.exit(1)
-
-attempts = 0
-dogtag_uri='ldap://localhost:%d' % DEFAULT_DSPORT
-updated = False
-
-while attempts < 10:
- conn = None
- try:
- conn = ldap2(shared_instance=False, ldap_uri=dogtag_uri)
- conn.connect(bind_dn=DN(('cn', 'directory manager')), bind_pw=dm_password)
- (entry_dn, entry_attrs) = conn.get_entry(dn, ['usercertificate'], normalize=False)
- entry_attrs['usercertificate'].append(cert)
- entry_attrs['description'] = '2;%d;%s;%s' % (serial_number, issuer, subject)
- conn.update_entry(dn, entry_attrs, normalize=False)
- updated = True
- break
- except errors.NetworkError:
- syslog.syslog(syslog.LOG_ERR, 'Connection to %s failed, sleeping 30s' % dogtag_uri)
- time.sleep(30)
- attempts += 1
- except errors.EmptyModlist:
- updated = True
- break
- except Exception, e:
- syslog.syslog(syslog.LOG_ERR, 'Updating agent entry failed: %s' % e)
- break
- finally:
- if conn.isconnected():
- conn.disconnect()
-
-if not updated:
- syslog.syslog(syslog.LOG_ERR, '%s: Giving up. This script may be safely re-executed.' % sys.argv[0])
- sys.exit(1)
+update_people_entry('ipara', dercert)
attempts = 0
updated = False
@@ -104,11 +60,11 @@ while attempts < 10:
conn.connect(ccache=ccache)
try:
(entry_dn, entry_attrs) = conn.get_entry(dn, ['usercertificate'])
- entry_attrs['usercertificate'] = cert
+ entry_attrs['usercertificate'] = dercert
conn.update_entry(dn, entry_attrs, normalize=False)
except errors.NotFound:
entry_attrs = dict(objectclass=['top', 'pkiuser', 'nscontainer'],
- usercertificate=cert)
+ usercertificate=dercert)
conn.add_entry(dn, entry_attrs, normalize=False)
except errors.EmptyModlist:
pass
diff --git a/ipapython/certmonger.py b/ipapython/certmonger.py
index f29050ea96318a9564358f42405fc8849a2e27b7..829e12e1016a886535b7e6f0c1c1207170c723c2 100644
--- a/ipapython/certmonger.py
+++ b/ipapython/certmonger.py
@@ -386,20 +386,22 @@ def dogtag_start_tracking(ca, nickname, pin, pinfile, secdir, command):
if not cert_exists(nickname, os.path.abspath(secdir)):
raise RuntimeError('Nickname "%s" doesn\'t exist in NSS database "%s"' % (nickname, secdir))
- if command is not None and not os.path.isabs(command):
- if sys.maxsize > 2**32:
- libpath = 'lib64'
- else:
- libpath = 'lib'
- command = '/usr/%s/ipa/certmonger/%s' % (libpath, command)
-
args = ["/usr/bin/getcert", "start-tracking",
"-d", os.path.abspath(secdir),
"-n", nickname,
"-c", ca,
- "-C", command,
]
+ if command is not None:
+ if not os.path.isabs(command):
+ if sys.maxsize > 2**32:
+ libpath = 'lib64'
+ else:
+ libpath = 'lib'
+ command = '/usr/%s/ipa/certmonger/%s' % (libpath, command)
+ args.append("-C")
+ args.append(command)
+
if pinfile:
args.append("-p")
args.append(pinfile)
diff --git a/ipaserver/install/cainstance.py b/ipaserver/install/cainstance.py
index e7b63f81e0df7b18881d523b92f711afd8c8c28a..ccf5298900479cc2e59dc22cbd71b7453d2479dc 100644
--- a/ipaserver/install/cainstance.py
+++ b/ipaserver/install/cainstance.py
@@ -35,11 +35,13 @@ import urllib
import xml.dom.minidom
import stat
import socket
+import syslog
import ConfigParser
from ipapython import dogtag
from ipapython.certdb import get_ca_nickname
from ipapython import certmonger
from ipalib import pkcs10, x509
+from ipalib import errors
from ipapython.dn import DN
import subprocess
import traceback
@@ -1533,11 +1535,16 @@ class CAInstance(service.Service):
'Unable to determine PIN for CA instance: %s' % str(e))
def track_servercert(self):
+ """
+ Specifically do not tell certmonger to restart the CA. This will be
+ done by the renewal script, renew_ca_cert once all the subsystem
+ certificates are renewed.
+ """
pin = self.__get_ca_pin()
certmonger.dogtag_start_tracking(
'dogtag-ipa-renew-agent', 'Server-Cert cert-pki-ca', pin, None,
self.dogtag_constants.ALIAS_DIR,
- 'restart_pkicad "Server-Cert cert-pki-ca"')
+ None)
def configure_renewal(self):
cmonger = ipaservices.knownservices.certmonger
@@ -1860,6 +1867,67 @@ def update_cert_config(nickname, cert):
base64.b64encode(cert),
quotes=False, separator='=')
+def update_people_entry(uid, dercert):
+ """
+ Update the userCerticate for an entry in the dogtag ou=People. This
+ is needed when a certificate is renewed.
+
+ uid: uid of user to update
+ dercert: An X509.3 certificate in DER format
+
+ Logging is done via syslog
+
+ Returns True or False
+ """
+ dn = DN(('uid',uid),('ou','People'),('o','ipaca'))
+ serial_number = x509.get_serial_number(dercert, datatype=x509.DER)
+ subject = x509.get_subject(dercert, datatype=x509.DER)
+ issuer = x509.get_issuer(dercert, datatype=x509.DER)
+
+ attempts = 0
+ dogtag_uri='ldap://localhost:%d' % DEFAULT_DSPORT
+ updated = False
+
+ try:
+ dm_password = certmonger.get_pin('internaldb')
+ except IOError, e:
+ syslog.syslog(syslog.LOG_ERR, 'Unable to determine PIN for CA instance: %s' % e)
+ return False
+
+ while attempts < 10:
+ conn = None
+ try:
+ conn = ldap2.ldap2(shared_instance=False, ldap_uri=dogtag_uri)
+ conn.connect(bind_dn=DN(('cn', 'directory manager')),
+ bind_pw=dm_password)
+ (entry_dn, entry_attrs) = conn.get_entry(dn, ['usercertificate'],
+ normalize=False)
+ entry_attrs['usercertificate'].append(dercert)
+ entry_attrs['description'] = '2;%d;%s;%s' % (serial_number, issuer,
+ subject)
+ conn.update_entry(dn, entry_attrs, normalize=False)
+ updated = True
+ break
+ except errors.NetworkError:
+ syslog.syslog(syslog.LOG_ERR, 'Connection to %s failed, sleeping 30s' % dogtag_uri)
+ time.sleep(30)
+ attempts += 1
+ except errors.EmptyModlist:
+ updated = True
+ break
+ except Exception, e:
+ syslog.syslog(syslog.LOG_ERR, 'Updating %s entry failed: %s' % (str(dn), e))
+ break
+ finally:
+ if conn.isconnected():
+ conn.disconnect()
+
+ if not updated:
+ syslog.syslog(syslog.LOG_ERR, 'Update failed.')
+ return False
+
+ return True
+
if __name__ == "__main__":
standard_logging_setup("install.log")
if not dogtag.install_constants.SHARED_DB:
--
1.8.0.2
_______________________________________________
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel
