Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
After a bunch more troubleshooting I finally have logs that are error free on
all 4 servers :-)
I couldn't find anything really useful on Google about this particular error :
attrlist_replace - attr_replace (nsslapd-referral,
ldap://ipadc.mydomain.net:389/o%3Dipaca) failed
So I am going to write about my experiences fixing it. There was a clue in a
thread here :
https://www.redhat.com/archives/freeipa-users/2015-March/msg00699.html
But if you are like me and chose FreeIPA because you wanted to spend your time
managing a lot of computers without worrying about the gorry technical details
of 389 directory server, the answer given in that thread needs some explaining.
On every domain controller in your network run this command :
ldapsearch -D "cn=directory manager" -W -b "o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
nscpentrywsi
In the output on each server, look for the following key : It tells you the
server's current ID :
nscpentrywsi: nsDS5ReplicaId: 1195
Now look for the ruv entries that look like this :
nscpentrywsi: nsds50ruv: {replica 1195 ldap://dc1-ipa-dev-nvan.mydomain
.net:389} 569afd7c04ab 569b5b0e04ab
Any of those ruvs that have an id number after the word replica need to be
deleted if the number doesn't match the number of one of your servers. They
are old entries from previously deleted agreements. Don't delete any that your
servers identified themselves current as though, that will crash the server.
Use the following ldap query to delete the old ones (where 21 in CLEANRUV21 is
the id number of the agreement you want to delete) :
ldapmodify -x -D "cn=directory manager" -W < dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV21
> EOF
I noticed more strange behavior here because even after I deleted every old
RUV, one of them came back all by itself. I assumed it must be part of an
agreement somewhere in the system and was getting re-created automatically so I
went hunting for more info. I noticed that the amount of unique servers listed
in the error log message on each server uniquely matched the number of maxcsn
entries in the ldap output of the tombstone search on each server. The entries
looked like this :
nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;dc2-ipa-dev-van.mydomain.net-to-
dc1-ipa-dev-nvan.mydomain.net;dc1-ipa-dev-nvan.mydomain.net;389
;unavailable
nscpentrywsi: nsds5agmtmaxcsn:
o=ipaca;masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat;dc2-ipa-dev-nvan.mydomain.net;389;1095;569ae
e5a00030038
I could tell by looking at the unavailable it meant it was having trouble
getting a csn number, but I didn't know how to delete them safely with ldap
syntax. Luckily, the new 4.3.0 interface calls these maxcsn entries segments.
Removing them using the web ui is kind of round about, but works eventually.
On each server, go to the web ui and one at a time delete and re-create all
segments in the ca topology USING THE TEXT BASED ONE, NOT THE GRAPHIC ONE (this
requires domain level 1). The reason this works is because the command to
delete a domain level segment also doubles as a command to clean local segments
that are still in the old local part of the ldap tree from domain level 0.
You still have to repeat it on each server (which is kind of funny because you
are deleting the domain level objects multiple times, but only because you need
to cause the local trigger on each server).
I noticed that after re-creation the names of the maxcsn entries in that ldap
query result are much more uniform. There are no 'masterAgreement' csn types,
all member servers that are not the CA master have no entries at all, even
after replication, and on the master, they are all labelled with the -to-
syntax instead of the pki syntax. I also noticed that some of my old invalid
agreements had the same server name on both sides of the -to- and now they all
perfectly match the segment names in the web ui.
I'm assuming all the bugs in 4.1.4 and 4.2.0 and 4.2.3 created a lot of garbage
entries.
Luckily, with the tools in 4.3.0 those can all be removed.
I have now been staring at logs that have zero errors for over 30 minutes, and
I was previously getting hundreds per second.
Although this is great news for me, it is not great news for anyone stuck on a
CentOS or RHEL machine with no upgrade path to 4.3.0 without switching to
Fedora who is experiencing the category of bugs (there were definitely multiple
ones) that I encountered trying to fix these replication issues.
-Original Message-
From: Nathan Peters
Sent: January-17-16 1:10 AM
To: Nathan Peters
Cc: freeipa-users@redhat.com
Subject: RE: [Freeipa-users] Replication failing on FreeIPA 4.2.0
After some amount of work, I was able to get my system ba
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
After some amount of work, I was able to get my system back to a state where it seems to be replicating ok, but not with FreeIPA 4.2.0. Because this was a production system with several hundred users and computers attached to it, a wipe of the domain was not an option so I decided to chance that the new replication topology features would help. I replaced each CentOS 7 domain controller with a Fedora 23 FreeIPA 4.2.3 host and while doing so I noticed an odd behavior of the RUVs. I know about the current bug where deleting a replica doesn't delete its RUV and I experienced that. I would run a command like this : dn: cn=clean 4, cn=cleanallruv, cn=tasks, cn=config objectclass: top objectclass: extensibleObject replica-base-dn: dc=mydomain,dc=net replica-id: 4 replica-force-cleaning: yes cn: clean 4 It would fail only if I was not in a current agreement with the new Fedora RUV for that host. Ie, if the old CentOS host had a RUV of 4, and the new Fedora host 15, and I was in an agreement with 15, that ldap code would delete 4, but if I was not in an agreement with 15, it would fail. After A while I had every server in an agreement with all others and got all the old RUVs cleared. I was still experiencing strange error messages in my logs with FreeIPA 4.2.3 so I decided to go all the way to 4.3.0. Here are the 4.2.3 errors : [16/Jan/2016:22:29:12 -0800] NSMMReplicationPlugin - replica_replace_ruv_tombstone: failed to update replication update vector for replica dc=mydomain,dc=net: LDAP error - 53 [16/Jan/2016:22:29:13 -0800] NSMMReplicationPlugin - agmt_delete: begin [16/Jan/2016:22:32:51 -0800] slapi_ldap_bind - Error: could not bind id [cn=Replication Manager masterAgreement1-dc2-ipa-dev-van.mydomain.net-pki-tomcat,ou=csusers,cn=config] authentication mechanism [SIMPLE]: error 32 (No such object) errno 0 (Success) On 4 servers, 3 upgrades to 4.3.0 went smooth, and 1 just hung during the %post section of the dnf install for an hour with ns-lapd process taking 100% cpu on all 4 cores until I stopped it. A subsequent ipa-server-upgrade fixed everything. With the new replication topology management graphs and controls in the ui, I was able to find some missing segments and replace some that were for some reason only 1 way. Replication seems to actually be proceeding smoothly and now instead of getting the hundreds of error log entries per second that I had reported in my earlier posts, I am only getting about 3 every 5 minutes. The bugs that were present in 4.2.0 and 4.2.3 seem to be almost entirely gone. I have ran the new topology suffix verification commands and they say everything is ok. I still get these errors in batches of 3, but they don't seem to be doing anything harmful in terms of my systems ability to operating and replicate properly : [17/Jan/2016:01:07:27 -0800] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-nvan.mydomain.net:389/o%3Dipaca) failed. -Original Message- From: freeipa-users-boun...@redhat.com [mailto:freeipa-users-boun...@redhat.com] On Behalf Of Nathan Peters Sent: January-15-16 10:00 AM To: Ludwig Krispenz Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 No dice on the rebuild and RUV cleaning. I'm still getting a pile of these on dc1-van : [15/Jan/2016:17:55:25 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): Skipping update operation with no message_id (uniqueid 6e6784a0-b5c911e5-b1f1cd78-f19552bb, CSN 569932db0004): I'm also getting these on dc1-nvan: [15/Jan/2016:17:45:36 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. -Original Message- From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: January-15-16 12:19 AM To: Nathan Peters Cc: Rob Crittenden; freeipa-users@redhat.com Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 On 01/15/2016 08:32 AM, Nathan Peters wrote: > I think I've finally started to make some progress on this. I did a lot of > googling and found some stuff to run manually in 389 ds through ldapmodify > commands to clean RUVs. During this process the server crashed and when it > came back online, suddenly all my ghost RUVs were visible through > ipa-replica-manage list-ruv. It was really strange, I had like 5 of them > from winsync agreements that kept failing and needing re-initialization, and > another 5 from my earlier re-installations of the 2 other domain controllers. > > I ran some more ruv cleanup commands through ldap and they all appear to be > gone. I'm not sure how the crash suddenly made them visible though or why > they had to be cleaned through ldapmodify directly and ipa-replica-manage > could neither see nor clean them. After a crash the RUV could be rebuil
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
No dice on the rebuild and RUV cleaning. I'm still getting a pile of these on dc1-van : [15/Jan/2016:17:55:25 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): Skipping update operation with no message_id (uniqueid 6e6784a0-b5c911e5-b1f1cd78-f19552bb, CSN 569932db0004): I'm also getting these on dc1-nvan: [15/Jan/2016:17:45:36 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. -Original Message- From: Ludwig Krispenz [mailto:lkris...@redhat.com] Sent: January-15-16 12:19 AM To: Nathan Peters Cc: Rob Crittenden; freeipa-users@redhat.com Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 On 01/15/2016 08:32 AM, Nathan Peters wrote: > I think I've finally started to make some progress on this. I did a lot of > googling and found some stuff to run manually in 389 ds through ldapmodify > commands to clean RUVs. During this process the server crashed and when it > came back online, suddenly all my ghost RUVs were visible through > ipa-replica-manage list-ruv. It was really strange, I had like 5 of them > from winsync agreements that kept failing and needing re-initialization, and > another 5 from my earlier re-installations of the 2 other domain controllers. > > I ran some more ruv cleanup commands through ldap and they all appear to be > gone. I'm not sure how the crash suddenly made them visible though or why > they had to be cleaned through ldapmodify directly and ipa-replica-manage > could neither see nor clean them. After a crash the RUV could be rebuilt from the changelog, and the changelog could contain references to cleaned ReplicaIds and so they came to live again. The cleanallruv task was enhanced to also clean the changelog, but this fix is in 1.3.4.2+. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
ywsi: nsruvReplicaLastModified: {replica 86} 5698802a
nscpentrywsi: nsds5ReplicaChangeCount: 908
nscpentrywsi: nsds5replicareapactive: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory
manager" -W < with scope subtree
# filter:
(&(objectclass=nstombstone)(nsUniqueId=---))
# requesting: nscpentrywsi
#
# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115061052Z
nscpentrywsi: nsState:: YADXiphWAgABAA
==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b0060
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
t:389} 569719a00060 56988ad90060
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
et:389} 569719a40061 569719e600110061
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
-mydomain.net:389} 56988ad7
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
v-mydomain.net:389}
nscpentrywsi: nsds5ReplicaChangeCount: 430
nscpentrywsi: nsds5replicareapactive: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]#
ldapsearch -xLLL -D "cn=directory manager" -W -b dc=dev-mydomain,dc=net \
'(&(nsuniqueid=---)(objectclass=nstombstone))'
ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 7
cn: clean 7
ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 5
cn: clean 5
dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 8
cn: clean 8
dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 6
cn: clean 6
dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 3
cn: clean 3
dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 9
cn: clean 9
dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 10
cn: clean 10
dn: cn=clean 86, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
replica-id: 86
cn: clean 86
-Original Message-
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Nathan Peters
Sent: January-14-16 8:25 PM
To: Rob Crittenden; Ludwig Krispenz; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
And the saga continues...
In my latest round of trying to fix this, I've attempted to remove the replicas
again, this time ensuring to use the --force and --cleanup flags to try to
remove the data. As you can see from the output below, it seems like every
possible error that could happen did. Some examples :
Ruvs needed to be manually cleaned.
Ldapsearch reveals that nothing at all has been deleted in the ruv section, and
I still have 6 duplicates somehow
ipa : ERRORInstance removal failed.
ipa : ERRORFailed to remove DS instance. You may need to remove
instance data manually
SASL failures while removing or trying to get replication agreements
At this point I think I may need to manually clean all the old data, but I'm
not even sure where to start.
Also... When dc1 is alone with no replicas, why does he have a ruv for
himself... does he need one ?
And... isn't there supposed to be some kind of clean-all-ruv task or is that
not in 4.2.0 but only a later version ?
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
< with scope subtree
# filter:
(&(objectclass=nstombstone)(nsUniqueId=---))
# requesting: nscpentrywsi
#
# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115061052Z
nscpentrywsi: nsState:: YADXiphWAgABAA
==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b0060
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
t:389} 569719a00060 56988ad90060
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
et:389} 569719a40061 569719e600110061
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
-mydomain.net:389} 56988ad7
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
v-mydomain.net:389}
nscpentrywsi: nsds5ReplicaChangeCount: 430
nscpentrywsi: nsds5replicareapactive: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@dc1-ipa-dev-van slapd-DEV-mydomain-NET]#
ldapsearch -xLLL -D "cn=directory manager" -W -b dc=dev-mydomain,dc=net \
'(&(nsuniqueid=---)(objectclass=nstombstone))'
ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 7
cn: clean 7
ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 5
cn: clean 5
dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 8
cn: clean 8
dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 6
cn: clean 6
dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 3
cn: clean 3
dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 9
cn: clean 9
dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 10
cn: clean 10
dn: cn=clean 86, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
replica-id: 86
cn: clean 86
-Original Message-
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Nathan Peters
Sent: January-14-16 8:25 PM
To: Rob Crittenden; Ludwig Krispenz; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
And the saga continues...
In my latest round of trying to fix this, I've attempted to remove the replicas
again, this time ensuring to use the --force and --cleanup flags to try to
remove the data. As you can see from the output below, it seems like every
possible error that could happen did. Some examples :
Ruvs needed to be manually cleaned.
Ldapsearch reveals that nothing at all has been deleted in the ruv section, and
I still have 6 duplicates somehow
ipa : ERRORInstance removal failed.
ipa : ERRORFailed to remove DS instance. You may need to remove
instance data manually
SASL failures while removing or trying to get replication agreements
At this point I think I may need to manually clean all the old data, but I'm
not even sure where to start.
Also... When dc1 is alone with no replicas, why does he have a ruv for
himself... does he need one ?
And... isn't there supposed to be some kind of clean-all-ruv task or is that
not in 4.2.0 but only a later version ?
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
g: nscpentrywsi
#
# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115040015Z
nscpentrywsi: nsState:: YAC3bphWAgACAA
==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b0060
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389}
569719a00060 56986eb90060
nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389}
56976b31004c 56976b5c0002004c
nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389}
5697661a0051 56986b550051
nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389}
569761d20056 5697620b00050056
nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389}
56973856005b 569738790004005b
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389}
569719a40061 569719e600110061
nscpentrywsi: nsruvReplicaLastModified: {replica 96
ldap://dc1-ipa-dev-van.mydomain.net:389} 56986eb7
nscpentrywsi: nsruvReplicaLastModified: {replica 76
ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
nscpentrywsi: nsruvReplicaLastModified: {replica 81
ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
nscpentrywsi: nsruvReplicaLastModified: {replica 86
ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
nscpentrywsi: nsruvReplicaLastModified: {replica 91
ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
nscpentrywsi: nsruvReplicaLastModified: {replica 97
ldap://dc1-ipa-dev-nvan.mydomain.net:389} 0000
nscpentrywsi: nsds5ReplicaChangeCount: 1465
nscpentrywsi: nsds5replicareapactive: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@dc1-ipa-dev-van slapd-mydomain-NET]#
-Original Message-
From: Nathan Peters
Sent: January-14-16 1:26 PM
To: Nathan Peters; Rob Crittenden; Ludwig Krispenz; freeipa-users@redhat.com
Subject: RE: [Freeipa-users] Replication failing on FreeIPA 4.2.0
So after some more forum searching I found a command that searches your ldap
database for RUVs. The output does not seems to match the list-ruv command for
each server. Is this where the issue lies in the database? I see 6 ruvs for
each host in the ldapsearch but only 3 in the ipa-replica-manage list-ruv
command
DC1-IPA-DEV-VAN OUTPUT
==
[root@dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager"
-W -b "o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=---))"
nscpentrywsiEnter LDAP Password:
# extended LDIF
#
# LDAPv3
# base with scope subtree
# filter:
(&(objectclass=nstombstone)(nsUniqueId=---))
# requesting: nscpentrywsi
#
# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication
Plugin,cn=plugins,cn=config
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160114210015Z
nscpentrywsi: nsState:: YABPDJhWAgACAA==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: numSubordinates: 2
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b0060
nscpentrywsi: nsds
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
dc1-ipa-dev-van.mydomain.net:389} 56980fca
nscpentrywsi: nsruvReplicaLastModified: {replica 81
ldap://dc1-ipa-dev-nvan.mydomain.net:389}
nscpentrywsi: nsruvReplicaLastModified: {replica 86
ldap://dc1-ipa-dev-nvan.mydomain.net:389}
nscpentrywsi: nsruvReplicaLastModified: {replica 91
ldap://dc2-ipa-dev-nvan.mydomain.net:389}
nscpentrywsi: nsruvReplicaLastModified: {replica 97
ldap://dc1-ipa-dev-nvan.mydomain.net:389}
nscpentrywsi: nsds5ReplicaChangeCount: 322
nscpentrywsi: nsds5replicareapactive: 0
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-replica-manage list-ruv
dc2-ipa-dev-nvan.mydomain.net:389: 10
dc1-ipa-dev-van.mydomain.net:389: 4
dc1-ipa-dev-nvan.mydomain.net:389: 9
-Original Message-
From: freeipa-users-boun...@redhat.com
[mailto:freeipa-users-boun...@redhat.com] On Behalf Of Nathan Peters
Sent: January-14-16 12:53 PM
To: Rob Crittenden; Ludwig Krispenz; freeipa-users@redhat.com
Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
I'm beginning to suspect there may be something wrong with my ldap database.
I actually completed deleted dc1-nvan and dc2-nvan last night, leaving only
dc1-van. I then re-provosioned dc1-nvan and dc2-nvan from scratch (os install
and everything).
After re-provisioning I was finally able to make a 3 way replication agreement
so each server was replicating with 2 others.
When I left, all servers were reporting successful output similar to this :
[root@dc2-ipa-dev-nvan ~]# ipa-replica-manage list -v `hostname`
p11-kit: ipa.p11-kit: x-public-key-info: invalid or unsupported attribute
dc1-ipa-dev-nvan.mydomain.net: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update
started
last update ended: 1970-01-01 00:00:00+00:00
dc1-ipa-dev-van.mydomain.net: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update
started
last update ended: 1970-01-01 00:00:00+00:00
When I came in this morning 8 hours later the logs are full of errors again:
So I have a few questions:
1)Is there any way to effectively 'clean' an ldap database?
2)Are there any commands I can run to find out if it is something in my
database that is causing issues?
-for troubleshooting this one I tried doing ruv-clean after I deleted my
replicas but it claimed their IDs no longer existed, so it thought they were
deleted properly.
3)Why even with successful replication are they still showing 1970 dates? I
never understand why they keep going back to that. They were at 2016 dates
last night...
Here are the error logs from each server :
===
Errors in dc1-nvan
===
[14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:27:36 +] NSMMReplicationPlugin - replication keep alive
entry already exists
[14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:33:43 +] NSMMReplicationPlugin - replication keep alive
entry already exists
[14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
===
Errors in dc2-nvan
===
[14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:26:11 +] NSMMReplicationPlugin - replication keep alive
entry already exists
[14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed.
[14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral,
ldap://dc1-i
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
I'm beginning to suspect there may be something wrong with my ldap database. I actually completed deleted dc1-nvan and dc2-nvan last night, leaving only dc1-van. I then re-provosioned dc1-nvan and dc2-nvan from scratch (os install and everything). After re-provisioning I was finally able to make a 3 way replication agreement so each server was replicating with 2 others. When I left, all servers were reporting successful output similar to this : [root@dc2-ipa-dev-nvan ~]# ipa-replica-manage list -v `hostname` p11-kit: ipa.p11-kit: x-public-key-info: invalid or unsupported attribute dc1-ipa-dev-nvan.mydomain.net: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 dc1-ipa-dev-van.mydomain.net: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: 0 Replica acquired successfully: Incremental update started last update ended: 1970-01-01 00:00:00+00:00 When I came in this morning 8 hours later the logs are full of errors again: So I have a few questions: 1)Is there any way to effectively 'clean' an ldap database? 2)Are there any commands I can run to find out if it is something in my database that is causing issues? -for troubleshooting this one I tried doing ruv-clean after I deleted my replicas but it claimed their IDs no longer existed, so it thought they were deleted properly. 3)Why even with successful replication are they still showing 1970 dates? I never understand why they keep going back to that. They were at 2016 dates last night... Here are the error logs from each server : === Errors in dc1-nvan === [14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:27:36 +] NSMMReplicationPlugin - replication keep alive entry already exists [14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:29:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:33:43 +] NSMMReplicationPlugin - replication keep alive entry already exists [14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:34:51 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. === Errors in dc2-nvan === [14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:19:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:26:11 +] NSMMReplicationPlugin - replication keep alive entry already exists [14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:29:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:31:46 +] NSMMReplicationPlugin - replication keep alive entry already exists [14/Jan/2016:20:34:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:34:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. [14/Jan/2016:20:34:52 +] attrlist_replace - attr_replace (nsslapd-referral, ldap://dc1-ipa-dev-van.mydomain.net:389/o%3Dipaca) failed. === Errors in dc1-van === [14/Jan/2016:20:31:45 +] NSMMReplicationPlugin - changelog program - _cl5GetDBFile: found DB object 7f5595d590d0 for database /var/lib/dirsrv/slapd-mydomain.net/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb0004.db [14/Jan/2016:20:31:45 +] NSMMReplicationPlugin - changelog program - cl5GetOperationCount: found DB object 7f5595d590d0 [14/Jan/2016:20:31:45 +] NSMMReplicationPlugin - changelog program - _cl5GetDBFile: found DB object 7f5595d590d0 for database /var/li
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
Nathan Peters wrote: > This just keeps on getting better and better. > > > > I need this replication working properly because it has caused about 7 > or 8 builds to fail today alone so I decided to just be done with > troubleshooting and remove the server from the domain and re-initialize it. > > > > I deleted it with ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net > and then removed then ran an ipa-server uninstall. I then made a new > gpg file for it on dc1-van and added it back as a replica. > > > > After I did that, I wanted to connect all 3 servers together and when I > run ipa-replica-manage connect on dc2-nvan I get this now. Im not sure > how troubleshoot that. > > > > > > dc1-ipa-dev-nvan.mydomain.net is an IPA Server, but it might be unknown, > foreign or previously deleted one. It means that the new server isn't showing up in the list of masters on dc2-nvan which points to continuing replication issues. rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 plus ldapmodify freezes up
On 01/12/2016 06:16 PM, Nathan Peters wrote: [12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): replay_update: Sending modify operation (dn="fqdn=mole2-mh-interopsnap1-nvan.mydomain.net,cn=computers,cn=accounts,dc=mydomain,dc=net" csn=569589060004) [12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): replay_update: modifys operation (dn="fqdn=mole2-mh-interopsnap1-nvan.mydomain.net,cn=computers,cn=accounts,dc=mydomain,dc=net" csn=569589060004)*not sent - empty* [12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): replay_update: Consumer successfully sent operation with csn 569589060004 [12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): Skipping update operation with no message_id (uniqueid 5a395106-b42a11e5-b6d1a094-64a60b74, CSN 569589060004): There is a series of updates like above that all have empty modifications (modifications that have been striped and are now empty) so it never sends those "empty" updates. Replication then keeps trying to send this same series of operations over and over. But it's not finding any updates in the changelog that are not stripped. So, can you make an update to entry (change a password, add a description attribute, whatever) and see what the logging shows and if that update replicates? Grep for "agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" and check the timestamps. I'm also only seeing issues when updates going to "dc1-ipa-dev-nvan:389", other replication agreements seem fine and accept the updates. Can any of the other replicas update dc1? Also, you can ignore: [12/Jan/2016:04:20:23 +] NSMMReplicationPlugin - replication keep alive entry already exists These messages were not supposed to be logged by default. Mark -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 plus ldapmodify freezes up
Ok. I did that and it ended properly. Debugging was enabled properly.
Here are the logs from dc1 where it is refusing the update ? Not sure how to
parse these...
[12/Jan/2016:23:11:15 +] NSMMReplicationPlugin - ruv_add_csn_inprogress:
successfully inserted csn 569560240005 into pending list
[12/Jan/2016:23:11:15 +] NSMMReplicationPlugin - conn=5219 op=121512
csn=569560240005 process postop: canceling operation csn
[12/Jan/2016:23:11:17 +] NSMMReplicationPlugin - ruv_add_csn_inprogress:
successfully inserted csn 569563c20005 into pending list
[12/Jan/2016:23:11:17 +] NSMMReplicationPlugin - conn=5219 op=121513
csn=569563c20005 process postop: canceling operation csn
[12/Jan/2016:23:11:19 +] NSMMReplicationPlugin - ruv_add_csn_inprogress:
successfully inserted csn 5695667c0005 into pending list
[12/Jan/2016:23:11:19 +] NSMMReplicationPlugin - conn=5219 op=121514
csn=5695667c0005 process postop: canceling operation csn
[12/Jan/2016:23:11:21 +] NSMMReplicationPlugin - ruv_add_csn_inprogress:
successfully inserted csn 569568660005 into pending list
[12/Jan/2016:23:11:21 +] NSMMReplicationPlugin - conn=5219 op=121515
csn=569568660005 process postop: canceling operation csn
[12/Jan/2016:23:11:23 +] - _csngen_adjust_local_time: gen state before
569589070003:1452640271:0:248
[12/Jan/2016:23:11:23 +] - _csngen_adjust_local_time: gen state after
56958913:1452640283:0:248
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - ruv_add_csn_inprogress:
successfully inserted csn 569589130004 into pending list
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - Purged state information
from entry
fqdn=proxy2-pr-prqa1-van.mydomain.net,cn=computers,cn=accounts,dc=mydomain,dc=net
up to CSN 568c4e8600040004
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - changelog program -
_cl5GetDBFileByReplicaName: found DB object 7fc743a5f4c0 for database
/var/lib/dirsrv/slapd-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb0004.db
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - changelog program -
_cl5GetDBFileByReplicaName: found DB object 7fc743a5f4c0 for database
/var/lib/dirsrv/slapd-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb0004.db
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - ruv_update_ruv:
successfully committed csn 569589130004
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin -
agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): State:
wait_for_changes -> wait_for_changes
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin -
agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): State:
wait_for_changes -> ready_to_acquire_replica
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin -
agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389): Cancelling
linger on the connection
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - windows sync -
agmt="cn=meToofficedc2.office.myotherdomain.net" (officedc2:389): State:
wait_for_changes -> wait_for_changes
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - windows sync -
agmt="cn=meToofficedc2.office.myotherdomain.net" (officedc2:389): State:
wait_for_changes -> ready_to_acquire_replica
[12/Jan/2016:23:11:23 +] - acquire_replica, supplier RUV:
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - supplier:
{replicageneration} 553fe9bb0004
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - supplier: {replica 4
ldap://dc1-ipa-dev-van.mydomain.net:389} 553fe9c90004
569589130004 5695881b
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - supplier: {replica 5
ldap://dc2-ipa-dev-nvan.mydomain.net:389} 5692120500010005
56947ab700020005 56947a54
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - supplier: {replica 3
ldap://dc1-ipa-dev-nvan.mydomain.net:389} 553fe9c40003
5694907b0003 56948f81
[12/Jan/2016:23:11:23 +] - acquire_replica, consumer RUV:
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - consumer:
{replicageneration} 553fe9bb0004
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - consumer: {replica 4
ldap://dc1-ipa-dev-van.mydomain.net:389} 553fe9c90004
5695890600040004 5695880e
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - consumer: {replica 3
ldap://dc1-ipa-dev-nvan.mydomain.net:389} 553fe9c40003
5694907b0003 56948f81
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - consumer: {replica 5}
5692120500010005 56947ab700020005 56947a54
[12/Jan/2016:23:11:23 +] - acquire_replica, supplier RUV is newer
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin - windows sync -
agmt="cn=meToofficedc2.office.myotherdomain.net" (officedc2:389): Cancelling
linger on the connection
[12/Jan/2016:23:11:23 +] NSMMReplicationPlugin -
agmt="cn=meTodc1-ipa-dev-nvan.mydomain.net" (dc1-ipa-dev-nvan:389):
Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0 plus ldapmodify freezes up
Nathan Peters wrote: > (I apologize if this isnt threading properly, I signed up with another > email address since my primary ISP is having issues right now) > > > > So to recap about the issues in this thread : > https://www.redhat.com/archives/freeipa-users/2016-January/msg00139.html > > > > I have 3 dcs. Dc1 and 3 replicate fine. Dc2 will replicate after a > re-initialize for about 12 hours, then start failing. > > > > Here are the logs from dc2. Strangely enough, I couldnt even turn the > debugging on properly. > > > > Here is what happens when I turn it on : > > > > ldapmodify -x -D "cn=directory manager" -w password > > dn: cn=config > > changetype: modify > > replace: nsslapd-errorlog-level > > nsslapd-errorlog-level: 8192 > > > > Modifying : cn=config > > ^C > > - > > And then it just freezes up indefinitely until I ctrl-c it. Strangely > enough, it appears to have actually made the modification because my log > is full of stuff now, but replication is still failing : > Use ^D to finish up ldapmodify. It may not be hanging, it may just be waiting on input. For 389-ds hangs you should see http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
