Rob, I wound up hitting the *nuclear option* button, and as you mentioned, removed 03 and rebuilt as a replica from 04.
I was still getting errors when attempting it regarding STARTTLS. Add this to your notes, and then feel free to slap me in the back of the head, but.. A huge part of our compliance / security team barking, is alllllllllllll about TLS. After hours of troubleshooting, I went into the "error" log /var/log/dirsrv/slapd-INSTANCE/error ... and I saw errors about my ciphers in dse.ldif. I had (a week or two ago) updated our ciphers listed under nsSSL3Ciphers: to disallow some ciphers that were considered weak now. My scans for TLS came back clean, but the error was because I left "default" there, instead of changing to "+all" Once I switched it to +all, the ipa-replica-install worked. Now I need to figure out how to get the webgui working and I'll be golden, I think! On Fri, Apr 14, 2023 at 11:07 AM Rob Crittenden <[email protected]> wrote: > Justen Long wrote: > > Rob, > > > > Currently only hiipa03 and hiipa04 (these are the masters/IPA servers) > > would utilize the ipa-server-certinstall commands. The other servers > > don't use any HTTPD or anything like that, and have never had issues in > > the past simply replacing the cert file in /etc/pki/tls/certs and > > letting it ride. > > > > So, with regard to the catch-22... > > > > hiipa03 isn't communicating appropriately or something, in my opinion. > > When attempting to run "ipa-certupdate" on it, it wants the Kerberos > > credentials (assuming 'kinit admin'). I run 'kinit admin" and enter the > > admin password that works on hiipa04, and it doesn't like it. I > > attempted to change that password using the long 'ldappasswd' command, > > but forgot to put dc=hilab,dc=ipa, and instead had put dc=hilab.ipa. > > This results in a failure, object not found.. (had a duh moment).. went > > to try the 'ldappasswd' command again, using dc=hilab,dc=ipa.. it then > > fails stating "Result: Operations error (1) \\ Additional info: Failed > > to update password". > > > > So, I just, for s&g, run 'kinit admin' again. Prompts for password. Try > > the one that failed because of the wrong dc input, and it takes it, > > stating its expired. Try to change it, and get the same "Result: > > Operations error (1) \\ Additional info:" Failed to update password." > > > > On the majority of the clients, when I enter "ipa-certupdate" it points > > to hiipa03, rather than hiipa04.. and fails because hiipa03 is still in > > shambles. I think if I can get hiipa03 back to operational, then > > everything else would work. > > > > 1) Is there some magical way to destroy the admin password (i.e. blank > > it out) so it prompts for a new one, or to replicate the admin stuff > > from hiipa04 to hiipa03 so that it all works? > > 2) If not, how can I possibly move forward? The only hold up is the > > admin password at this point. > > It's hard to say without any logs. It sure seems like an issue with the > 389 server on 03. It isn't clear what you've already done, like > restarting services. I assume you got the certificates replaced there as > well? > > But if passwords can't be set that points to something in 389. The > errors log might contain information. > > The hardcore option would be to uninstall IPA from 03 and install it as > a new replica from scratch. This assumes that 04 really is working > properly. > > rob > > > > > On Fri, Apr 14, 2023 at 7:11 AM Rob Crittenden <[email protected] > > <mailto:[email protected]>> wrote: > > > > Justen Long wrote: > > > Rob, > > > > > > FreeIPA documentation stated that all clients must receive the > > > ipa-updatecert command before installing the new server cert.. on > the > > > hiipa04 server I installed the cert in a hurry for validity sake. > Does > > > that mess with anything? > > > > ipa-certupdate distributes the CA chain. If the chain didn't change > then > > it should be fine. > > > > > > > > Note: hiipa03 seems to be primary but they’re both masters. I > > can’t get > > > the ipa-server-certinstall to take on this server stating it’s > missing > > > the whole chain.. but it worked just fine on hiipa04.. so that’s > why I > > > was trying to “kinit admin” on 03, to run ipa-updatecert or the > > > ipa-cacert-manage there if it didn’t take it from 04, which I’m > > assuming > > > it didn’t. > > > > You could try going back in time and use ipa-cacert-manage install to > > update the chain and then run ipa-server-certinstall. > > > > But like I said before, if you're changing your chain AND replacing > the > > certs at the same time none of your clients are going to know what > > to do. > > > > The process is: > > > > 1. ipa-cacert-manage install <new chain> > > 2. ipa-certupdate on ALL enrolled machines > > 3. Replace the server certs > > > > If you deviate then you're caught in the catch-22 I mentioned. > Clients > > won't trust the updated chain in order to download the updated chain. > > The aren't a lot of great options out of that. Unenrolling and > > re-enrolling the client is the best way. > > > > rob > > > > > > On Thu, Apr 13, 2023 at 2:41 PM Rob Crittenden > > <[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > > > Justen Long wrote: > > > > One go back.. when I tried to run "ipa-certupdate" on the > > other hosts > > > > (clients), it points to hiipa03 and fails for SSL still.. > > so, hoping > > > > once I get the cert updated on THAT server, that all > > restores to its > > > > okay state. > > > > > > You shouldn't need to go back for clients. Now that the server > > > certificates are valid they should just work. Unless you also > > changed > > > the CA chain in which case you're caught in a catch-22: you > > need to get > > > the updated chain from the server but you don't trust the > > chain of the > > > server. > > > > > > rob > > > > > > > > > > > On Thu, Apr 13, 2023 at 1:46 PM Justen Long > > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>>> > > > wrote: > > > > > > > > Quick update and answers to your questions. > > > > > > > > We have two IPA servers, both masters, hiipa03 and > hiipa04. > > > > > > > > hiipa04, I was able to set the time back, run the > > > ipa-ca-cert-manage > > > > and update the CA (minimally, but enough to accept the > > new cert).. > > > > ipa-certupdate ran fine on it. Then, ran > > ipa-server-certinstall on > > > > it, and it took. Website loads, can log into it, do some > > user > > > > management stuff.. can't run "ipa-replica-manage list" > > as it kicks > > > > an error for sslv3 handshake failure.. but was trying > > that to > > > remove > > > > hiipa03 and copy 04 to 03 somehow, maybe? > > > > > > > > hiipa03 is still giving me grief. Set the time using > > timedatectl, > > > > verified ntp is off and 'date' reports properly. I had > > tried to > > > > follow > > > > > > > > > this: > https://computingforgeeks.com/reset-freeipa-admin-password-as-root-user-on-linux/ > > > > when it wasn't accepting the admin password.. it failed > > saying > > > > object not found. So, I try 'kinit admin' again, with > > the new > > > > password I tried.. says its expired. Type it in, type in > > a new > > > > password.. and then it failed saying "kinit: Password > change > > > failed > > > > while getting initial credentials" > > > > > > > > On Thu, Apr 13, 2023 at 1:40 PM Rob Crittenden > > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>> wrote: > > > > > > > > Justen Long wrote: > > > > > I'm getting closer... it's not recognizing my > > admin password > > > > for IPA, or > > > > > for my personal account with admin rights now.. > > but no more > > > > SSL errors.. > > > > > just can't run ipa-certupdate without the proper > > > kerberos creds.. > > > > > > > > By not recognizing your password I assume you mean > > kinit is > > > > failing? Is > > > > the KDC running? I assume 389-ds is running? All > > restarted > > > after > > > > time > > > > became stable in the past? > > > > > > > > rob > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 12:51 PM Justen Long > > > > <[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>> > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>>> wrote: > > > > > > > > > > Following up, I see the date command just > > changed it > > > > momentarily... > > > > > using timedatectl and will report back. > > > > > > > > > > On Thu, Apr 13, 2023 at 12:31 PM Justen Long > > > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>> <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>>> wrote: > > > > > > > > > > Rob, > > > > > > > > > > I entered 'date --date="7 April 2023", > > verified it > > > > updated the > > > > > system time appropriately. Restarted > dirsrv, > > > ipa-custodia, > > > > > ipa-otpd, httpd.. krb5kdc and kadmin > > failed. Still, > > > > tried to > > > > > send ipa cert-update, and it popped the > > same SSL > > > > Certificate > > > > > Verify Failed error. > > > > > > > > > > On Thu, Apr 13, 2023 at 11:32 AM Rob > > Crittenden > > > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> > > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>>> wrote: > > > > > > > > > > Justen Long wrote: > > > > > > Additionally, is there any way to > > force the CA > > > > cert update > > > > > to be > > > > > > recognized? When I run it to update > > the CA > > > chain, > > > > > everything is > > > > > > verified.. but /etc/ipa/ca.crt > > didn't reflect > > > > the change.. > > > > > so I manually > > > > > > populated it by copying over the > guts of > > > the CA > > > > bundle to the > > > > > > /etc/ipa/ca.crt before trying to > install > > > the new > > > > server > > > > > cert and it > > > > > > still doesn't recognize it as > > trusted although > > > > the issuer > > > > > is the same > > > > > > and within the CA bundle. > > > > > > > > > > This is going to sound weird, but I'd > just > > > go back > > > > in time > > > > > to April 10, > > > > > restart all services but ntp (which > will > > > reset the > > > > time) and > > > > > then the > > > > > commands should work. Once the certs > are > > > updated and > > > > > working, return to > > > > > present time. > > > > > > > > > > rob > > > > > > > > > > > > > > > > > On Thu, Apr 13, 2023 at 6:20 AM > > Justen Long > > > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>> > > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>>> > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto: > [email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>>>>> wrote: > > > > > > > > > > > > Rob, > > > > > > > > > > > > Apologies for the delay in > > response. Once > > > > I'm home, I > > > > > don't have > > > > > > access to the information readily > > > available > > > > to respond > > > > > with. Here is > > > > > > the information you requested: > > > > > > > > > > > > The version of IPA we are using > is > > > 4.6.8, rpm > > > > > specifically for us is > > > > > > > > > ipa-server-4.6.8-5.el7.centos.12.x86_64 and > > > > we are > > > > > using CentOS 7.9 > > > > > > currently with plans to move to > > RHEL9 > > > within > > > > the next > > > > > year or so. > > > > > > > > > > > > Unfortunately, 'ipa config-show' > > doesn't > > > > work. It > > > > > populates the same > > > > > > error stating "ipa: ERROR: cannot > > > connect to > > > > > > 'https://ipaServer/ipa/json': > [SSL: > > > > > CERTIFICATE_VERIFY_FAILED] > > > > > > certificate verify failed > > (_ssl.c:618). > > > > > > > > > > The smack heard around the world was > > my head > > > > hitting my > > > > > desk. Of course > > > > > this command failed. > > > > > > > > > > > > > > > > > We have ~50 hosts connected via > IPA. > > > We have > > > > two IPA > > > > > servers, one as > > > > > > a replica of the other. > > > > > > > > > > > > 'getcert list' only shows 1 > > certificate. > > > > It's state is > > > > > "MONITORING" > > > > > > and seems related to kerberos. > > > > > > > > > > > > As far as I know, we don't use > IPA > > > CA-issued > > > > > certificates. I recall > > > > > > seeing errors yesterday stating > > CA wasn't > > > > enabled on > > > > > our servers. We > > > > > > have always used 3rd party CAs > to my > > > knowledge. > > > > > > > > > > > > -justen > > > > > > > > > > > > On Wed, Apr 12, 2023 at 2:42 PM > Rob > > > Crittenden > > > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>> > > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>> > > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>>> > > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>> > > > > > <mailto:[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>> > > > > <mailto:[email protected] <mailto: > [email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>>>>> > > wrote: > > > > > > > > > > > > Justen Long via > > FreeIPA-users wrote: > > > > > > > Thanks in advance for your > > replies.. > > > > I've spent > > > > > 7 hours > > > > > > looking through posts here > > and trying > > > > > everything... I'm stuck. > > > > > > > > > > > > > > Background: I am a System > > > > Administrator in a closed, > > > > > > classified environment. > > > Unfortunately, I > > > > cannot > > > > > post logging > > > > > > here, but I can refer to > them as > > > needed. > > > > > > > > > > > > > > I inherited this system > from > > > someone who > > > > > departed the program > > > > > > a year or so ago. Fast > > forward to > > > today, the > > > > > server certs > > > > > > expired yesterday. > > Admittedly, I'm > > > > unfamiliar (or > > > > > was) with the > > > > > > certificate update process > > for IPA > > > > servers. On a > > > > > typical server, > > > > > > we replace the old cert and > > > restart the > > > > httpd > > > > > services; however, > > > > > > I realize this cannot work > > with IPA > > > > servers now. > > > > > > > > > > > > > > Additionally to all of > > this, the > > > CA chain > > > > > updated 6 months ago. > > > > > > > > > > > > > > I ran ipa-cacert-manage to > > > update the > > > > CA chain. > > > > > When trying to > > > > > > run ipa-certupdate, I > received > > > errors for an > > > > > invalid server > > > > > > certificate (it expired on > > 11 April > > > > 2023). It > > > > > simply won't > > > > > > connect to the web server. > HTTPD > > > failed > > > > as well, > > > > > so I had to add > > > > > > "NSSEnforceValidCerts off" > > to the > > > > nss.conf file > > > > > for HTTPD to > > > > > > start. Still, no dice. > > > > > > > > > > > > > > I've ran > > ipa-server-certinstall for > > > > the new > > > > > cert/key as well, > > > > > > and it fails saying its not > > > trusted ("Peer's > > > > > certificate issuer > > > > > > is not trusted [certutil: > > > certificate is > > > > invalid: > > > > > Peer's > > > > > > Certificate issuer is not > > recognized] > > > > Please run > > > > > > ipa-cacert-manage install and > > > > ipa-certupdate to > > > > > install the CA > > > > > > certificate.... which, as > > reported > > > > above, can't > > > > > complete. > > > > > > > > > > > > > > I'm at a total loss > > here... and > > > really > > > > > struggling being new to > > > > > > all this and trying my best > > to keep it > > > > afloat. Any > > > > > help would be > > > > > > GREATLY appreciated! > > > > > > > > > > > > Let's gather some > > information first. > > > > > > > > > > > > What version of IPA is this, > > on what > > > > distribution? > > > > > > > > > > > > IPA designates one server to > be > > > the "renewal > > > > > master" which > > > > > > handles the > > > > > > renewals. The output of `ipa > > > > config-show` should > > > > > tell you > > > > > > (depending on > > > > > > version). That's the server > you > > > want to > > > > work on. > > > > > > > > > > > > How many servers in your > > topology and > > > > how many > > > > > have a CA installed? > > > > > > > > > > > > Does `getcert list` show a > > set of 8-10 > > > > tracked > > > > > certificates? > > > > > > What are > > > > > > the states? > > > > > > > > > > > > You mention > > > ipa-server-certinstall. Are > > > > you using > > > > > 3rd party > > > > ��> > certificates > > > > > > in addition to IPA CA-issued > > > > certificates or was > > > > > that just an > > > > > > attempt to > > > > > > get things working again? > > > > > > > > > > > > rob > > > > > > > > > > > > > > > > > > > > > >
_______________________________________________ FreeIPA-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
