[Freeipa-users] IPA Replication Status

2014-07-23 Thread Choudhury, Suhail
Hi,

I'm finding that on all IPA servers in 1 cluster the replication status shows 
as either busy or started, but no succeeded status is being reported:

[root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
recsds1.bskyb.com: replica
  last init status: None
  last init ended: None
  last update status: 1 Can't acquire busy replica
  last update ended: 2014-07-23 11:29:48+00:00
recsds3.bskyb.com: replica
  last init status: None
  last init ended: None
  last update status: 1 Can't acquire busy replica
  last update ended: 2014-07-23 11:29:46+00:00
[root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
recsds1.bskyb.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
started
  last update ended: None
recsds3.bskyb.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
started
  last update ended: None

This is as opposed to another IPA cluster:

/home/sch # ipa-replica-manage list -v $HOSTNAME
ipa01.ath.skycdc.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
succeeded
  last update ended: 2014-07-23 11:24:22+00:00
ipa01.hhe.skycdc.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
succeeded
  last update ended: 2014-07-23 11:24:22+00:00
ipa02.ath.skycdc.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
succeeded
  last update ended: 2014-07-23 11:24:22+00:00
ipa02.hhe.skycdc.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
succeeded
  last update ended: 2014-07-23 11:24:22+00:00
ipa02.slu.skycdc.com: replica
  last init status: None
  last init ended: None
  last update status: 0 Replica acquired successfully: Incremental update 
succeeded
  last update ended: 2014-07-23 11:24:21+00:00

Do you know what may be causing this?

Regards,
Suhail Choudhury.
DevOps | Recommendations Team | BSkyB

Information in this email including any attachments may be privileged, 
confidential and is intended exclusively for the addressee. The views expressed 
may not be official policy, but the personal views of the originator. If you 
have received it in error, please notify the sender by return e-mail and delete 
it from your system. You should not reproduce, distribute, store, retransmit, 
use or disclose its contents to anyone. Please note we reserve the right to 
monitor all e-mail communication through our internal and external networks. 
SKY and the SKY marks are trademarks of British Sky Broadcasting Group plc and 
Sky International AG and are used under licence. British Sky Broadcasting 
Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration 
No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) 
are direct or indirect subsidiaries of British Sky Broadcasting Group plc 
(Registration No. 2247735). All of the companies mentioned in this paragraph 
are incorporated in England and Wales and share the same registered office at 
Grant Way, Isleworth, Middlesex TW7 5QD.
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] IPA Replication Status

2014-07-23 Thread Martin Kosek
On 07/23/2014 01:36 PM, Choudhury, Suhail wrote:
 Hi,
 
 I'm finding that on all IPA servers in 1 cluster the replication status shows 
 as either busy or started, but no succeeded status is being reported:
 
 [root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
 recsds1.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 1 Can't acquire busy replica
   last update ended: 2014-07-23 11:29:48+00:00
 recsds3.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 1 Can't acquire busy replica
   last update ended: 2014-07-23 11:29:46+00:00
 [root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
 recsds1.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 started
   last update ended: None
 recsds3.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 started
   last update ended: None
 
 This is as opposed to another IPA cluster:
 
 /home/sch # ipa-replica-manage list -v $HOSTNAME
 ipa01.ath.skycdc.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 succeeded
   last update ended: 2014-07-23 11:24:22+00:00
 ipa01.hhe.skycdc.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 succeeded
   last update ended: 2014-07-23 11:24:22+00:00
 ipa02.ath.skycdc.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 succeeded
   last update ended: 2014-07-23 11:24:22+00:00
 ipa02.hhe.skycdc.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 succeeded
   last update ended: 2014-07-23 11:24:22+00:00
 ipa02.slu.skycdc.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 succeeded
   last update ended: 2014-07-23 11:24:21+00:00
 
 Do you know what may be causing this?

It is difficult to advise with just this information, you would need to check
for errors in DS log (/var/log/dirsrv/slapd-YOUR-REALM/errors).

Martin

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project


Re: [Freeipa-users] IPA Replication Status

2014-07-23 Thread Choudhury, Suhail
I have the following errors on different boxes:

[root@recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Replicas 
have not been cleaned yet, retrying in 10 seconds
[23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Waiting 
for all the replicas to finish cleaning...
[23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 10 seconds
[23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 20 seconds
[23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 40 seconds

[root@recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:10 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The 
consumer may need to be reinitialized.
[23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds2.bskyb.com (recsds2:389): changelog iteration code 
returned a dummy entry with csn 53c7c6b100020010, skipping ...
[23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds4.bskyb.com (recsds4:389) - 
Can't locate CSN 53c7ba7500040010 in the changelog (DB rc=-30988). The 
consumer may need to be reinitialized.
[23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): changelog iteration code 
returned a dummy entry with csn 53c7c6b100020010, skipping ...
[23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The 
consumer may need to be reinitialized.

[root@recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
[23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
[23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com

[root@recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 53c7ba7e00030010): 
Server is unwilling to perform (53). Will retry later.
[23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 53c7ba900010): 
Server is unwilling to perform (53). Will retry later.
[23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 53c7ba7500050010): 
Server is unwilling to perform (53). Will retry later.

The background to this is a storage crash caused the master CA IAP to get 
fudged, and I then proceeded to promote a replica to master CA, re-added 
crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.

Regards,
Suhail Choudhury.
DevOps | Recommendations Team | BSkyB



From: Martin Kosek [mko...@redhat.com]
Sent: 23 July 2014 12:43
To: Choudhury, Suhail; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] IPA Replication Status

On 07/23/2014 01:36 PM, Choudhury, Suhail wrote:
 Hi,

 I'm finding that on all IPA servers in 1 cluster the replication status shows 
 as either busy or started, but no succeeded status is being reported:

 [root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
 recsds1.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 1 Can't acquire busy replica
   last update ended: 2014-07-23 11:29:48+00:00
 recsds3.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 1 Can't acquire busy replica
   last update ended: 2014-07-23 11:29:46+00:00
 [root@recsds2 ~]# ipa-replica-manage list -v $HOSTNAME
 recsds1.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 started
   last update ended: None
 recsds3.bskyb.com: replica
   last init status: None
   last init ended: None
   last update status: 0 Replica acquired successfully: Incremental update 
 started
   last update ended: None

 This is as opposed

Re: [Freeipa-users] IPA Replication Status

2014-07-23 Thread Martin Kosek
On 07/23/2014 01:58 PM, Choudhury, Suhail wrote:
 I have the following errors on different boxes:
 
 [root@recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
 [23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: 
 Replicas have not been cleaned yet, retrying in 10 seconds
 [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: 
 Waiting for all the replicas to finish cleaning...
 [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not 
 all replicas finished cleaning, retrying in 10 seconds
 [23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not 
 all replicas finished cleaning, retrying in 20 seconds
 [23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not 
 all replicas finished cleaning, retrying in 40 seconds
 
 [root@recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
 [23/Jul/2014:12:52:10 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
 Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The 
 consumer may need to be reinitialized.
 [23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - 
 agmt=cn=meTorecsds2.bskyb.com (recsds2:389): changelog iteration code 
 returned a dummy entry with csn 53c7c6b100020010, skipping ...
 [23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds4.bskyb.com (recsds4:389) - 
 Can't locate CSN 53c7ba7500040010 in the changelog (DB rc=-30988). The 
 consumer may need to be reinitialized.
 [23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - 
 agmt=cn=meTorecsds4.bskyb.com (recsds4:389): changelog iteration code 
 returned a dummy entry with csn 53c7c6b100020010, skipping ...
 [23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
 Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The 
 consumer may need to be reinitialized.
 
 [root@recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
 [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
 entry 
 nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
 [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
 entry 
 nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
 [23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone 
 entry 
 nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
 
 [root@recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
 [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
 agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay 
 change (uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 
 53c7ba7e00030010): Server is unwilling to perform (53). Will retry later.
 [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
 agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay 
 change (uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 
 53c7ba900010): Server is unwilling to perform (53). Will retry later.
 [23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - 
 agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay 
 change (uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 
 53c7ba7500050010): Server is unwilling to perform (53). Will retry later.
 
 The background to this is a storage crash caused the master CA IAP to get 
 fudged, and I then proceeded to promote a replica to master CA, re-added 
 crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.
 
 Regards,
 Suhail Choudhury.
 DevOps | Recommendations Team | BSkyB

Somebody from DS may have a better idea, but it seems to me that the fastest
way to recover is to either ipa-replica-manage re-initialize the replicas
from the new CA IPA master (I am assuming this one is running more or less
fine) or even to uninstall, ipa-replica-manage del it and install again to
get a clean environment.

Martin

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project


Re: [Freeipa-users] IPA Replication Status

2014-07-23 Thread Rich Megginson

On 07/23/2014 06:02 AM, Martin Kosek wrote:

On 07/23/2014 01:58 PM, Choudhury, Suhail wrote:

I have the following errors on different boxes:

[root@recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Replicas 
have not been cleaned yet, retrying in 10 seconds
[23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Waiting 
for all the replicas to finish cleaning...
[23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 10 seconds
[23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 20 seconds
[23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all 
replicas finished cleaning, retrying in 40 seconds

[root@recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:10 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The consumer may 
need to be reinitialized.
[23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds2.bskyb.com (recsds2:389): changelog iteration code 
returned a dummy entry with csn 53c7c6b100020010, skipping ...
[23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds4.bskyb.com (recsds4:389) - 
Can't locate CSN 53c7ba7500040010 in the changelog (DB rc=-30988). The consumer may 
need to be reinitialized.
[23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): changelog iteration code 
returned a dummy entry with csn 53c7c6b100020010, skipping ...
[23/Jul/2014:12:52:13 +0100] agmt=cn=meTorecsds2.bskyb.com (recsds2:389) - 
Can't locate CSN 53c7ba270010 in the changelog (DB rc=-30988). The consumer may 
need to be reinitialized.

[root@recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
[23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
[23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone 
entry 
nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb@recs.bskyb.com,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com

[root@recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
[23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 53c7ba7e00030010): Server is 
unwilling to perform (53). Will retry later.
[23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 53c7ba900010): Server is 
unwilling to perform (53). Will retry later.
[23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - 
agmt=cn=meTorecsds4.bskyb.com (recsds4:389): Consumer failed to replay change 
(uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 53c7ba7500050010): Server is 
unwilling to perform (53). Will retry later.

The background to this is a storage crash caused the master CA IAP to get 
fudged, and I then proceeded to promote a replica to master CA, re-added 
crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.

Regards,
Suhail Choudhury.
DevOps | Recommendations Team | BSkyB

Somebody from DS may have a better idea, but it seems to me that the fastest
way to recover is to either ipa-replica-manage re-initialize the replicas
from the new CA IPA master (I am assuming this one is running more or less
fine) or even to uninstall, ipa-replica-manage del it and install again to
get a clean environment.


Try the re-initialize first.  That will be necessary since you have the 
following error: The consumer may need to be reinitialized.


Note that busy is a normal condition.  A consumer allows updates from 
only 1 supplier at a time, and the other suppliers will get a busy signal.


What version of 389-ds-base are you using?  rpm -q 389-ds-base



Martin



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project