Re: [Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake
David Copperfield wrote: Hi Rich and all, For the latest IPA version 2.1.3-9 on red hat 6.2, the CA RUV records clearance seems a must. Before clearance the annoying messages are filling /var/log/dirsrv/slapd-PKI-IPA/errors on master file, while after clearance the entries are gone. [16/May/2012:19:49:40 -0700] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=ipaca: 20 [16/May/2012:19:49:57 -0700] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=ipaca: 20 [16/May/2012:19:53:21 -0700] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=ipaca: 20 [16/May/2012:19:53:24 -0700] NSMMReplicationPlugin - repl_set_mtn_referrals: could not set referrals for replica o=ipaca: 20 Before clearing CA, in there error log file, there are entry list below, while after clearance it is gone too. [16/May/2012:19:49:21 -0700] NSMMReplicationPlugin - replica_check_for_data_reload: Warning: data for replica o=ipaca does not match the data in the changelog (replica data (4fb467560051) changelog (4fb467560051)). Recreating the changelog file. This could affect replication with replica's consumers in which case the consumers should be reinitialized. [16/May/2012:19:49:21 -0700] - slapd started. Listening on All Interfaces port 7389 for LDAP requests [16/May/2012:19:49:21 -0700] - Listening on All Interfaces port 7390 for LDAPS requests Hope in 2.2.0 we only need to clear user data type replication, and can safely ignore CA type which will automatically cleaned -- in sync with user type replication. The CA is just another 389-ds instance. It needs to be cleaned the same way any other instance would. Nothing will change in 2.2. Hopefully this will be available for the 3.0 release. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake
Sorry to declare success too quick, :( In fact, it is worse now, the IPA master fail after performing the above steps including the RUV cleaning. I've only one working replica and I'm afraid to do anything on it. On The IPA master, after I ran 'service ipa restart' it reported OK, but 'ipa user-find' failed. so I cleared my Kerboers TGT ticket, ran 'kinit admin' to try my luck, the IPA master failed with the following message, it showed that 389 port listening disappeared for unknown reasons. [root@ipamaster slapd-EXAMPLE-COM]# kinit admin kinit: Generic error (see e-text) while getting initial credentials [root@ipamaster slapd-EXAMPLE-COM]# netstat -antup | grep -i LISTEN | grep ns tcp 0 0 :::7389 :::* LISTEN 6550/ns-slapd tcp 0 0 :::7390 :::* LISTEN 6550/ns-slapd [root@ipamaster slapd-EXAMPLE-COM]# The error logs are pasted here too. [16/May/2012:14:41:43 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [16/May/2012:14:41:43 -0700] - slapd started. Listening on All Interfaces port 389 for LDAP requests [16/May/2012:14:41:43 -0700] - Listening on All Interfaces port 636 for LDAPS requests [16/May/2012:14:41:43 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests [16/May/2012:14:41:43 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:14:41:43 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:14:41:43 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:14:41:46 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth resumed Thanks. --David From: David Copperfield cao2...@yahoo.com To: JR Aquino jr.aqu...@citrix.com Cc: freeipa-users@redhat.com freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 1:23 PM Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake Hi JR, Thanks a lot! It works perfectly. The only extra thing probably goes with 2.1.3 only: I need to find and clear ghost RUV records for CA database, and remove it from master and all other live replicas as well. BTW, on 2.2.0 the two database backends still are separate, or merged into one? Thanks. --David From: JR Aquino jr.aqu...@citrix.com To: David Copperfield cao2...@yahoo.com Cc: FreeIPAUsers freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 12:57 PM Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake On May 16, 2012, at 12:23 PM, David Copperfield wrote: Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.com, but accidentally the mouse moved to ipareplica02.example.com and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa user-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# ipa host-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02 slapd-EXAMPLE-COM]# On the IPA master, It was found that ipareplica02 didn't show up in 'host-find' list or 'service-find' list. Though it still showed in the master list reported by 'ipa-replica-manage' and 'ipa-csreplica-manage', the real command 'ipa-replica-manage list ipareplica02' fails with LDAP could't reach error. What should I do now? Is there are any other ways to recover besides uninstall and reinstall of IPA replica ipareplica02? BTW, it will be more than appreciated if the web UI could pop up a warning prompt when
Re: [Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake
Try: ipactl stop then ipactl start Doesn't look like dirsrv is running on 389 and 636 ~ Jr Aquino | Sr. Information Security Specialist GIAC Certified Incident Handler | GIAC WebApp Penetration Tester Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117x-apple-data-detectors://0/0 T: +1 805.690.3478tel:+1%C2%A0805.690.3478 C: +1 805.717.0365tel:+1%20805.717.0365 jr.aqu...@citrixonline.commailto:jr.aqu...@citrixonline.com http://www.citrixonline.comhttp://www.citrixonline.com/ On May 16, 2012, at 2:54 PM, David Copperfield wrote: Sorry to declare success too quick, :( In fact, it is worse now, the IPA master fail after performing the above steps including the RUV cleaning. I've only one working replica and I'm afraid to do anything on it. On The IPA master, after I ran 'service ipa restart' it reported OK, but 'ipa user-find' failed. so I cleared my Kerboers TGT ticket, ran 'kinit admin' to try my luck, the IPA master failed with the following message, it showed that 389 port listening disappeared for unknown reasons. [root@ipamaster slapd-EXAMPLE-COM]# kinit admin kinit: Generic error (see e-text) while getting initial credentials [root@ipamaster slapd-EXAMPLE-COM]# netstat -antup | grep -i LISTEN | grep ns tcp0 0 :::7389 :::* LISTEN 6550/ns-slapd tcp0 0 :::7390 :::* LISTEN 6550/ns-slapd [root@ipamaster slapd-EXAMPLE-COM]# The error logs are pasted here too. [16/May/2012:14:41:43 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.commailto:ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) [16/May/2012:14:41:43 -0700] - slapd started. Listening on All Interfaces port 389 for LDAP requests [16/May/2012:14:41:43 -0700] - Listening on All Interfaces port 636 for LDAPS requests [16/May/2012:14:41:43 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests [16/May/2012:14:41:43 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:14:41:43 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:14:41:43 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:14:41:46 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth resumed Thanks. --David From: David Copperfield cao2...@yahoo.commailto:cao2...@yahoo.com To: JR Aquino jr.aqu...@citrix.commailto:jr.aqu...@citrix.com Cc: freeipa-users@redhat.commailto:freeipa-users@redhat.com freeipa-users@redhat.commailto:freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 1:23 PM Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake Hi JR, Thanks a lot! It works perfectly. The only extra thing probably goes with 2.1.3 only: I need to find and clear ghost RUV records for CA database, and remove it from master and all other live replicas as well. BTW, on 2.2.0 the two database backends still are separate, or merged into one? Thanks. --David From: JR Aquino jr.aqu...@citrix.commailto:jr.aqu...@citrix.com To: David Copperfield cao2...@yahoo.commailto:cao2...@yahoo.com Cc: FreeIPAUsers freeipa-users@redhat.commailto:freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 12:57 PM Subject: Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake On May 16, 2012, at 12:23 PM, David Copperfield wrote: Hi all, I accidentally removed one of my IPA replica host on IPA web UI by mistake, on the host list I planed to remove ipaclient02.example.comhttp://ipaclient02.example.com/, but accidentally the mouse moved to ipareplica02.example.comhttp://ipareplica02.example.com/ and the latter got removed without a prompt. I realized the mistake and tried to recover from this disaster but it was already too late, the change propagated to all the replicas and the poor ipareplica02 now stops functioning. [root@ipareplica02 slapd-EXAMPLE-COM]# ipa service-find ipa: ERROR: cannot connect to u'https://ipareplica02.qe9.jigsaw.com/ipa/xml': Internal Server Error [root@ipareplica02
Re: [Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake
Could that be because of removing ghost entries in CA database? Another possible place could be the deleting/clearing option itself. One annoying thing that I've found is: I cleared the RUV records from IPA servers one by one, then I restart IPA services on the servers one by one again, ldapsearch showed that the RUV ghost entries popped up again. :( I had to kill it again and again across the IPA server farms, then restart IPA servers one by one, check again, until the ghost RUV entries disappeared from all and didn't come back -- It is very, VERY exhausting and annoying. After that I still need to stop IPA replica first, then restart IPA master and until now it worked -- ipa commands and kinit worked. At last I brought up the valid replica and it worked this time as well. Now it was time to reinstall the failed IPA replica and it was installed and up and running well. After I tested with 'ipa user-add', 'ipa-user-delete' and found that the replication did work across the IPA master and IPA replicas. I tested the last time and found the following messages in the error log file on IPA master, it maybe harmless but I am not sure: [16/May/2012:16:18:36 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up[16/May/2012:16:18:36 -0700] schema-compat-plugin - warning: no entries set up under ou=SUDOers, dc=jigsaw,dc=com [16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition. [16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition. [16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - slapd started. Listening on All Interfaces port 389 for LDAP requests [16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - Listening on All Interfaces port 636 for LDAPS requests [16/May/2012:16:18:36 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests [16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica02.example.com (ipareplica02:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))[16/May/2012:16:18:39 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica02.example.com (ipareplica02:389): Replication bind with GSSAPI auth resumed [16/May/2012:16:18:39 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth resumed --David From: JR Aquino jr.aqu...@citrix.com To: David Copperfield cao2...@yahoo.com Cc: JR Aquino jr.aqu...@citrix.com; Rob Crittenden rcrit...@redhat.com; freeipa-users@redhat.com freeipa-users@redhat.com Sent: Wednesday, May 16, 2012 4:00 PM Subject: Re: Still not working -- Re: [Freeipa-users] What to do next???: IPA replica host entry is removed on web UI by mistake Try: ipactl stop then ipactl start Doesn't look like dirsrv is running on 389 and 636 ~ Jr Aquino | Sr. Information Security Specialist GIAC Certified Incident Handler | GIAC
Re: [Freeipa-users] Still not working -- Re: What to do next???: IPA replica host entry is removed on web UI by mistake
Whew, glad to hear you got through it! The 389 ds crew is working on making the cleanruv into an internal automated process. I empathize completely. The gssapi errors are generally benign. They come up because ldap starts before the kdc. Keeping your head in the cloud ~ Jr Aquino | Sr. Information Security Specialist GIAC Certified Incident Handler | GIAC WebApp Penetration Tester Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117 jr.aqu...@citrix.commailto:jr.aqu...@citrix.com http://www.citrixonline.com On May 16, 2012, at 4:29 PM, David Copperfield cao2...@yahoo.commailto:cao2...@yahoo.com wrote: Could that be because of removing ghost entries in CA database? Another possible place could be the deleting/clearing option itself. One annoying thing that I've found is: I cleared the RUV records from IPA servers one by one, then I restart IPA services on the servers one by one again, ldapsearch showed that the RUV ghost entries popped up again. :( I had to kill it again and again across the IPA server farms, then restart IPA servers one by one, check again, until the ghost RUV entries disappeared from all and didn't come back -- It is very, VERY exhausting and annoying. After that I still need to stop IPA replica first, then restart IPA master and until now it worked -- ipa commands and kinit worked. At last I brought up the valid replica and it worked this time as well. Now it was time to reinstall the failed IPA replica and it was installed and up and running well. After I tested with 'ipa user-add', 'ipa-user-delete' and found that the replication did work across the IPA master and IPA replicas. I tested the last time and found the following messages in the error log file on IPA master, it maybe harmless but I am not sure: [16/May/2012:16:18:36 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up[16/May/2012:16:18:36 -0700] schema-compat-plugin - warning: no entries set up under ou=SUDOers, dc=jigsaw,dc=com [16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition. [16/May/2012:16:18:36 -0700] - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=jigsaw,dc=com--no CoS Templates found, which should be added before the CoS Definition. [16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.commailto:ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - slapd started. Listening on All Interfaces port 389 for LDAP requests [16/May/2012:16:18:36 -0700] set_krb5_creds - Could not get initial credentials for principal [ldap/ipamaster.example@example.commailto:ldap/ipamaster.example@example.com] in keytab [WRFILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))[16/May/2012:16:18:36 -0700] - Listening on All Interfaces port 636 for LDAPS requests [16/May/2012:16:18:36 -0700] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI requests [16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica02.example.com (ipareplica02:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found)) [16/May/2012:16:18:36 -0700] slapi_ldap_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: error -2 (Local error) [16/May/2012:16:18:36 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica01.example.com (ipareplica01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Credentials cache file '/tmp/krb5cc_496' not found))[16/May/2012:16:18:39 -0700] NSMMReplicationPlugin - agmt=cn=meToipareplica02.example.com (ipareplica02:389): Replication bind with GSSAPI auth resumed [16/May/2012:16:18:39 -0700]
