[Freeipa-users] Re: Removal & clean up certificates from o=ipaca
On Tue, 7 Feb 2023 10:35:35 +0100
Florence Blanc-Renaud wrote:
> Hi,
>
> On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
> freeipa-users@lists.fedorahosted.org> wrote:
>
> > Hi David. I had the same issue here and found your writeup to be very
> > helpful.
> >
> > I used more or less the same ldap actions to delete the certificates
> > and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> > just the one "used"/"correct" certificate for the host, but the main
> > issue is not fixed. The webUI still displays all the old certificates
> > that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> > takes very long, around 1-2 minutes.
> >
> > So I want to know what else needs to be done to make the webUI "forget"
> > about the wrongly issued certificates?
> >
> > Where does the webUI get its list of certificates?
> > I did some searching through the code and could only find the JS
> > code that invokes a RPC call. But I could not find the code that
> > handles that call.
> >
> > The webui is making a call equivalent to "ipa cert-find" which is handled
> by the following code:
> https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L1496
>
> The call looks for certificates in multiple locations:
> - in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
> - in the suffix "dc=example,dc=com", in the users/hosts/services entries
>
> You cleaned the certificates from the cert repository but there may be many
> entries (users/hosts/services) containing a userCertificate attribute. To
> avoid seeing those certs you would have to delete the corresponding
> userCertificate values.
>
> HTH,
> flo
>
Thanks flo, this was exactly what I needed, I managed to delete them
from userCertificate under
"fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
and now the webui is fast again!
In retrospect I think making the webui work fast even with such a large
number of certificates would be better, since this cleanup is not easy
and it's possible to delete too much. That would probably mean adding a
pager and filtering to the certificate list which is displayed on the
host page. Right now they are all displayed at once on one page and
there is no way to filter out e.g. all revoked certificates. Also find
and fix why loading the hosts list takes so long even though it doesn't
display any certificates on that page.
Here are the commands to delete the certificates from userCertificate
of the host. Add them to those in the previous message, after o=ipaca
cleaning but before changelog purging.
First save a list of all current userCertificate entries to a file
ldapsearch -LLL -x -D "cn=directory manager" -W -o ldif-wrap=no
-b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
userCertificate > userCertificate_badhost.ldif
Copy the certificate you want to keep from where it's stored on the
host to where you are working. Put it in badhost.crt in PEM format.
Create a new ldif without that certificate.
sed -e '/^dn: .*$/d'
-e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-'|tr -d '\n')##"
userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif
You can delete other certificates that you also want to keep from this
ldif. If you know the serial numbers of them, you can construct a shell
one-liner script with a loop that iterates over each certificate in each
line of the file, uses openssl x509 to retrieve its serial from
base64-encoded certificate and deletes the certificate from the file if
it matches.
I didn't need that so I don't have the script.
Format the ldif so it can be consumed by ldapmodify. Note that you have
to remove the newlines that the mailing list adds so that everything is
on one line, especially printf's format strings!
{ printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\n
changetype: modify\ndelete: userCertificate\n';
while read -r; do if [ -z "$REPLY" ]; then continue; fi;
printf '%s\n' "$REPLY"; done < userCertificate_badhost_to_delete.ldif;
} > userCertificate_badhost_to_delete_ldapmodify.ldif
Test if the ldif is correct
ldapmodify -nv -x -D "cn=directory manager" -W
-f userCertificate_badhost_to_delete_ldapmodify.ldif
echo $? should say 0, if not, there's an error in the ldif you need to
correct.
Then remove the -nv and run the ldapmodify again to delete the entries
for real.
> >
> > IIRC my issue that caused certmonger to request a certificate over and
> > over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> > 16.04. The path to ca_external_helper changed but it was not changed in
> > /var/lib/certmonger/cas/* which caused certmonger to fail running
> > ca_external_helper. To fix it I did:
> >
> > sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> > /var/lib/certmonger/cas/*
> >
> >
> > Below is the exact procedure I used to delete the certificates from
> > LDAP.
> >
> > First fix the issue that caused the issuing of too many
[Freeipa-users] Re: Removal & clean up certificates from o=ipaca
Hi,
On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
freeipa-users@lists.fedorahosted.org> wrote:
> Hi David. I had the same issue here and found your writeup to be very
> helpful.
>
> I used more or less the same ldap actions to delete the certificates
> and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> just the one "used"/"correct" certificate for the host, but the main
> issue is not fixed. The webUI still displays all the old certificates
> that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> takes very long, around 1-2 minutes.
>
> So I want to know what else needs to be done to make the webUI "forget"
> about the wrongly issued certificates?
>
> Where does the webUI get its list of certificates?
> I did some searching through the code and could only find the JS
> code that invokes a RPC call. But I could not find the code that
> handles that call.
>
> The webui is making a call equivalent to "ipa cert-find" which is handled
by the following code:
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L1496
The call looks for certificates in multiple locations:
- in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
- in the suffix "dc=example,dc=com", in the users/hosts/services entries
You cleaned the certificates from the cert repository but there may be many
entries (users/hosts/services) containing a userCertificate attribute. To
avoid seeing those certs you would have to delete the corresponding
userCertificate values.
HTH,
flo
>
> IIRC my issue that caused certmonger to request a certificate over and
> over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> 16.04. The path to ca_external_helper changed but it was not changed in
> /var/lib/certmonger/cas/* which caused certmonger to fail running
> ca_external_helper. To fix it I did:
>
> sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> /var/lib/certmonger/cas/*
>
>
> Below is the exact procedure I used to delete the certificates from
> LDAP.
>
> First fix the issue that caused the issuing of too many certificates.
> Make sure it successfully issued and saved the cert on the client and
> that it's in status "MONITORING", "stuck: no".
> Find the serial number of the cert currently present on the client.
> 'sudo getcert list', look at "certificate:" in my case it was in
> "/etc/ssl/private/hostname-ipa-cert.crt"
> openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> In my case it was 268369940.
>
> Used the following shell script to revoke all the certificates with
> serial not matching. I did this before I knew howo to get the cert
> serials from ldap so it uses ipa cert-find. It's a slow process.
> for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s"
> ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
>
> You can view all the revoked cert cn's with this command before
> deleting them.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
>
> Make a list of all cert cn's not matching the used cert, save output
> into a file, ready to be read by ldapdelete later.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
> cert_to_delete_not_used_badhost
>
> Make a list of all the requestId for all the certs to be deleted.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
> 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
>
> In my case there were a couple more requests than issued certs, I'm not
> sure why. I made a list of all requests for this host excluding the
> requestId of the correct cert. First find the correct/used cert
> requestId. In my case it was 9990026.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> 'requestId:\K.*'
>
> Then get a list of all requests for that host, excluding that one
> requestId.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=ca,ou=requests,o=ipaca"
> '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
>
> Count the number of certs/requests from the previous operations. The
> first two must match, the 3rd shows how many extra requests there are.
> wc -l cert_to_delete_not_used_badhost
> cert_request_to_delete_not_used_from_metaInfo_badhost
> cert_request_to_delete_not_used_badhost
> 3982 cert_to_delete_not_used_badhost
> 3982 cert_request_to_delete_not_used_from_metaInfo_badhost
> 3990
[Freeipa-users] Re: Removal & clean up certificates from o=ipaca
Hi David. I had the same issue here and found your writeup to be very
helpful.
I used more or less the same ldap actions to delete the certificates
and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
just the one "used"/"correct" certificate for the host, but the main
issue is not fixed. The webUI still displays all the old certificates
that I have deleted from LDAP. Opening the "Hosts" tab or a host page
takes very long, around 1-2 minutes.
So I want to know what else needs to be done to make the webUI "forget"
about the wrongly issued certificates?
Where does the webUI get its list of certificates?
I did some searching through the code and could only find the JS
code that invokes a RPC call. But I could not find the code that
handles that call.
IIRC my issue that caused certmonger to request a certificate over and
over was caused by a bug after upgrading a client from Ubuntu 14.04 to
16.04. The path to ca_external_helper changed but it was not changed in
/var/lib/certmonger/cas/* which caused certmonger to fail running
ca_external_helper. To fix it I did:
sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
/var/lib/certmonger/cas/*
Below is the exact procedure I used to delete the certificates from
LDAP.
First fix the issue that caused the issuing of too many certificates.
Make sure it successfully issued and saved the cert on the client and
that it's in status "MONITORING", "stuck: no".
Find the serial number of the cert currently present on the client.
'sudo getcert list', look at "certificate:" in my case it was in
"/etc/ssl/private/hostname-ipa-cert.crt"
openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
In my case it was 268369940.
Used the following shell script to revoke all the certificates with
serial not matching. I did this before I knew howo to get the cert
serials from ldap so it uses ipa cert-find. It's a slow process.
for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk '/Serial
number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s" ]; then
continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
You can view all the revoked cert cn's with this command before
deleting them.
ldapsearch -LLL -x -D "cn=directory manager" -W -b
"ou=certificateRepository,ou=ca,o=ipaca"
'(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
Make a list of all cert cn's not matching the used cert, save output
into a file, ready to be read by ldapdelete later.
ldapsearch -LLL -x -D "cn=directory manager" -W -b
"ou=certificateRepository,ou=ca,o=ipaca"
'(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
cert_to_delete_not_used_badhost
Make a list of all the requestId for all the certs to be deleted.
ldapsearch -LLL -x -D "cn=directory manager" -W -b
"ou=certificateRepository,ou=ca,o=ipaca"
'(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
In my case there were a couple more requests than issued certs, I'm not
sure why. I made a list of all requests for this host excluding the
requestId of the correct cert. First find the correct/used cert
requestId. In my case it was 9990026.
ldapsearch -LLL -x -D "cn=directory manager" -W -b
"ou=certificateRepository,ou=ca,o=ipaca"
'(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP 'requestId:\K.*'
Then get a list of all requests for that host, excluding that one
requestId.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=ca,ou=requests,o=ipaca"
'(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
Count the number of certs/requests from the previous operations. The
first two must match, the 3rd shows how many extra requests there are.
wc -l cert_to_delete_not_used_badhost
cert_request_to_delete_not_used_from_metaInfo_badhost
cert_request_to_delete_not_used_badhost
3982 cert_to_delete_not_used_badhost
3982 cert_request_to_delete_not_used_from_metaInfo_badhost
3990 cert_request_to_delete_not_used_badhost
So there are 8 extra requests without corresponding certs. I chose to
ignore them for now. They're probably fine to delete in the future.
Before deleting the requests, make a file in a format ldapdelete
expects.
while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY"; done <
cert_request_to_delete_not_used_from_metaInfo_badhost >
cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
Now the actual deletion steps.
Delete the certs.
ldapdelete -x -D "cn=directory manager" -W -f cert_to_delete_not_used_badhost
Delete the requests.
ldapdelete -x -D "cn=directory manager" -W -f
cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
(you can add '-nv' to test ldapdelete)
After this, I decided to trim the changelog and tombstones from ldap.
Roughly followed the info from
[Freeipa-users] Re: Removal & clean up certificates from o=ipaca
Hello, I did request and certificate suppression test and restart IPA stack. It works! ldapdelete -x -D "cn=directory manager" -W "cn=87289,ou=ca,ou=requests,o=ipaca" ldapdelete -x -D "cn=directory manager" -W "cn=87273,ou=certificateRepository,ou=ca,o=ipaca" I am going to generate the list of request and certificate entries that are useless. Hereafter a little procedure: (cn (in ou=certificateRepository,ou=ca,o=ipaca) is equal to serialId decimal in x509 certificate) Backup IPA and save the ipaca tree (sudo ldapsearch -x -h localhost -D "cn=directory manager" -W -b o=ipaca > /var/lib/ipa/backup/all) Certificate tree purge (ou=certificateRepository,ou=ca,o=ipaca): 1. Identify entry that have to be excluded (non garbage certificate: used & expired certificates) - Get serial ID of certificate used: sudo openssl x509 -in xxx.crt -text -noout | grep "Seria\|Not\|Sub" 2. Get garbage certificate list (used & expired certificates are excluded): ldapsearch -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~=)(!(cn=))(certStatus=VALID))' dn | grep "cn=" | sed -e "s/dn: //" -e "/\#/d" > cert_ Request tree purge (ou=ca,ou=requests,o=ipaca): 1. Identify entry that have to be excluded (non garbage certificate: used & expired certificates) - Get requestID of certificate used: sudo ldapsearch -x -D "cn=directory manager" -W -b "cn=,ou=certificateRepository,ou=ca,o=ipaca" '(subjectName~=)' "metaInfo" - Get requestID of certificate expired: sudo ldapsearch -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~=)(!(certStatus=VALID)))' "metaInfo" 2. Get garbage certificate request list (used & expired certificates are excluded): sudo ldapsearch -x -D "cn=directory manager" -W -b "ou=ca,ou=requests,o=ipaca" '(&(extdata-req--005fsubject--005fname--002ecn=)(&(!(cn=))(!(cn=' dn | grep "cn=" | sed -e "s/dn: //" -e "/\#/d" > req_ Check that number of request and certificate entry to purge are equal: grep -c cn= cert_ grep -c cn= req_ (I hope this will help) Thank you for your response, - Original Message - From: "Fraser Tweedale" To: "freeipa-users" Cc: "David Goudet" Sent: Thursday, November 8, 2018 2:28:03 AM Subject: Re: [Freeipa-users] Removal & clean up certificates from o=ipaca On Wed, Nov 07, 2018 at 04:29:36PM +0100, David Goudet via FreeIPA-users wrote: > Hello all, > Hi David, > I have to clean up lot of useless certificate in dirsrv database. > Because of resubmit loop on Certmonger client, i have 99,9% of certificate in > dirsrv database that are useless and not obsolete (expiration in 2020) (it > represent ~85 000 certificates). > Did you already resolve the Certmonger resubmit loop? > These useless certificates produce some issues on FreeIPA: > - decrease FreeIPA performances on CLI and GUI > - increase the LDAP size > - increase size and time of FreeIPA backup > ... > > Is it possible to purge these certificates in dirsrv database and how? > Yes. You can remove them manually. > I found two branches in LDAP directory about these certificates: > > dn: cn=xxx,ou=ca,ou=requests,o=ipaca > dn: cn=yyy,ou=certificateRepository,ou=ca,o=ipaca > The certificateRepository contains the issued certificates, the ou=ca,ou=requests contains data about the certificate requests. Each certificateRepository entry contains a reference to the request that produced it. You'll have to manually work out which certs you don't want, delete its certificateRepository entry (cn is the serial number), and delete the corresponding request entry. > I can remove all requests and certificates entry from dirsrv > database but how it is supported by PKI manager Dogtag (CRL, > certificate generation, OCSP)? > CRLs and OCSP responses are generated using the data from the certificateRepository. Forgetting about non-expired certificates is not valid under X.509, but since you have an operational issue, just choose carefully which ones you keep and which ones you delete. Don't delete the entry for any certificates in active use, OR any non-expired but revoked certificate where you want it to appear in CRLs or want valid OCSP responses for that certificate. Also, whatever certificate has the highest serial number, do not delete it. When using sequential serial number (which is how Dogtag gets configured by FreeIPA) upon startup Dogtag looks for the highest serial number to work out what is the next serial number to use. So keep the cert with the highest serial number otherwise serial numbers will be re-used. Cheers, Fraser -- David GOUDET LYRA NETWORK IT Operations service Tel : +33 (0)5 32 09 09 74 | Poste : 574 ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines:
[Freeipa-users] Re: Removal & clean up certificates from o=ipaca
On Wed, Nov 07, 2018 at 04:29:36PM +0100, David Goudet via FreeIPA-users wrote: > Hello all, > Hi David, > I have to clean up lot of useless certificate in dirsrv database. > Because of resubmit loop on Certmonger client, i have 99,9% of certificate in > dirsrv database that are useless and not obsolete (expiration in 2020) (it > represent ~85 000 certificates). > Did you already resolve the Certmonger resubmit loop? > These useless certificates produce some issues on FreeIPA: > - decrease FreeIPA performances on CLI and GUI > - increase the LDAP size > - increase size and time of FreeIPA backup > ... > > Is it possible to purge these certificates in dirsrv database and how? > Yes. You can remove them manually. > I found two branches in LDAP directory about these certificates: > > dn: cn=xxx,ou=ca,ou=requests,o=ipaca > dn: cn=yyy,ou=certificateRepository,ou=ca,o=ipaca > The certificateRepository contains the issued certificates, the ou=ca,ou=requests contains data about the certificate requests. Each certificateRepository entry contains a reference to the request that produced it. You'll have to manually work out which certs you don't want, delete its certificateRepository entry (cn is the serial number), and delete the corresponding request entry. > I can remove all requests and certificates entry from dirsrv > database but how it is supported by PKI manager Dogtag (CRL, > certificate generation, OCSP)? > CRLs and OCSP responses are generated using the data from the certificateRepository. Forgetting about non-expired certificates is not valid under X.509, but since you have an operational issue, just choose carefully which ones you keep and which ones you delete. Don't delete the entry for any certificates in active use, OR any non-expired but revoked certificate where you want it to appear in CRLs or want valid OCSP responses for that certificate. Also, whatever certificate has the highest serial number, do not delete it. When using sequential serial number (which is how Dogtag gets configured by FreeIPA) upon startup Dogtag looks for the highest serial number to work out what is the next serial number to use. So keep the cert with the highest serial number otherwise serial numbers will be re-used. Cheers, Fraser ___ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
