Re: [Freeipa-users] WG: Re: Haunted servers?
Hi, I have one scenario where I can show the comeback of the "ghost" rids. but it requires a server where the rids have successfully cleaned and it is killed or crashes. In that case, if the "ghost" rids have not yet been trimmed from the changelog they can be recreated from information in the changelog. they can then also propagate to other servers Could something similar have happened in your environment ? Ludwig On 06/12/2015 07:38 AM, Christoph Kaminski wrote: I've been too early pleased :/ After ipactl restart of our first master (where we re-initialize from) are the 'ghost' rids again there... I think there is something like a fs backup for dirsrv (changelog?) but where? > > we had the same problem (and some more) and yesterday we have > successfully cleaned the gohst rid's > > our fix: > > 1. stop all cleanallruv Tasks, if it works with ipa-replica-manage > abort-clean-ruv. It hasnt worked here. We have done it manually on > ALL replicas with: > a) replica stop > b) delete all nsds5ReplicaClean from /etc/dirsrv/slapd-HSO/dse.ldif > c) replica start > > 2. prepare on EACH ipa a cleanruv ldif file with ALL ghost rids > inside (really ALL from all ipa replicas, we has had some rids only > on some replicas...) > Example: > > dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config > changetype: modify > replace: nsds5task > nsds5task:CLEANRUV11 > > dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config > changetype: modify > replace: nsds5task > nsds5task:CLEANRUV22 > > dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config > changetype: modify > replace: nsds5task > nsds5task:CLEANRUV37 > ... > > 3. do a "ldapmodify -h 127.0.0.1 -D "cn=Directory Manager" -W -x -f > $your-cleanruv-file.ldif" on all replicas AT THE SAME TIME :) we > used terminator for it (https://launchpad.net/terminator). You can > open multiple shell windows inside one window and send to all at the > same time the same commands... > > 4. we have done a re-initialize of each IPA from our first master > > 5. restart of all replicas > > we are not sure about the point 3 and 4. Maybe they are not > necessary, but we have done it. > > If something fails look at defect LDAP entries in whole ldap, we > have had some entries with 'nsunique-$HASH' after the 'normal' name. > We have deleted them. > > MfG > Christoph Kaminski > > -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] WG: Re: Haunted servers?
Hi, On 06/22/2015 09:48 AM, Christoph Kaminski wrote: > > from an earlier post it looks like they are from the o=ipaca > backend, did you clean the ruvs there ? we have only done a 'normal' cleanruv... How can I clean them there? either you try the cleanallruv: # ldapmodify -D "cn=directory manager" -W -a dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config objectclass: extensibleObject replica-base-dn:*o=ipaca* replica-id: 8 cn: clean 8 you have to set the replicabase dn. or, if you want to go to the method of running cleanruv individually on all servers you have to use the dn of the ipaca replica: > dn: cn=replica,cn=*o**\3Dipaca*,cn=mapping tree,cn=config > changetype: modify > replace: nsds5task > nsds5task:CLEANRUV11 > > to know which are the correct current rids for this backend you > could do on each active server a search for > ... -b "cn=config" "(&(objectclass=nsds5replica)( > nsDS5ReplicaRoot=o=ipaca))" nsDS5ReplicaId > > then you could search > > ldapsearch -h -D "cn=Directory Manager" -W -b "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=--- > ))" > to see what you have in the ruv and eventually clean them > On 06/19/2015 01:48 PM, Christoph Kaminski wrote: > Ludwig Krispenz schrieb am 19.06.2015 13:23:43: > > > > > > the first search is for the replication agreements and they keep > > > info about the consumer ruv, used in replication session. you cannot > > > modify these, but they are maintained in the dse.ldif, you could > > > edit the dse.ldif when the server is stopped. > > > > big thx, we try it and I let you know if it works! > > > Greetz -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] WG: Re: Haunted servers?
> > from an earlier post it looks like they are from the o=ipaca > backend, did you clean the ruvs there ? we have only done a 'normal' cleanruv... How can I clean them there? > > to know which are the correct current rids for this backend you > could do on each active server a search for > ... -b "cn=config" "(&(objectclass=nsds5replica)( > nsDS5ReplicaRoot=o=ipaca))" nsDS5ReplicaId > > then you could search > > ldapsearch -h -D "cn=Directory Manager" -W -b "o=ipaca" > "(&(objectclass=nstombstone)(nsUniqueId=--- > ))" > to see what you have in the ruv and eventually clean them > On 06/19/2015 01:48 PM, Christoph Kaminski wrote: > Ludwig Krispenz schrieb am 19.06.2015 13:23:43: > > > > > > the first search is for the replication agreements and they keep > > > info about the consumer ruv, used in replication session. you cannot > > > modify these, but they are maintained in the dse.ldif, you could > > > edit the dse.ldif when the server is stopped. > > > > big thx, we try it and I let you know if it works! > > > Greetz -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] WG: Re: Haunted servers?
Hi Christoph,
bad news. So to summarize, you have a procedure to cleanup your env, but
once you restart the master the ghosts are back.
I really want to find out where they are coming from, so If you have to
restart your server, could you please lookup these data, after the
server is stopped:
dbscan -f /var/lib/dirsrv/slapd-s/db/userRoot/nsuniqueid.db
-k =--- -r
=---
3
this gives you the RUVID and you can look it up in the database
[root@elkris scripts]# dbscan -f
/var/lib/dirsrv/slapd-/db/userRoot/id2entry.db -K
id 3
rdn: nsuniqueid=---
nsUniqueId: ---
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 51dc3bac0064
nsds50ruv: {replica 100 ldap://localhost:30522}
557fd5410064 557fd9d30
064
nsds50ruv: {replica 200 ldap://localhost:4945} 557fd6e600c8
557fda0e00
..
then check the contents of the changelog:
[root@elkris scripts]# dbscan -f
/var/lib/dirsrv/slapd-/changelogdb/ec450682-7c0a11e2-aa0e8005-8430f734_51dc3bac0064.db
| more
the first entries contain th ruv data:
dbid: 006f
entry count: 307
dbid: 00de
purge ruv:
{replicageneration} 51dc3bac0064
{replica 100 ldap://localhost:30522}
{replica 200 ldap://localhost:30522}
dbid: 014d
max ruv:
{replicageneration} 51dc3bac0064
{replica 100} 557fd5410064 557fd9d30064
{replica 200} 557fd6e600c8 557fda0e00c8
On 06/12/2015 07:38 AM, Christoph Kaminski wrote:
I've been too early pleased :/ After ipactl restart of our first
master (where we re-initialize from) are the 'ghost' rids again there...
I think there is something like a fs backup for dirsrv (changelog?)
but where?
>
> we had the same problem (and some more) and yesterday we have
> successfully cleaned the gohst rid's
>
> our fix:
>
> 1. stop all cleanallruv Tasks, if it works with ipa-replica-manage
> abort-clean-ruv. It hasnt worked here. We have done it manually on
> ALL replicas with:
> a) replica stop
> b) delete all nsds5ReplicaClean from /etc/dirsrv/slapd-HSO/dse.ldif
> c) replica start
>
> 2. prepare on EACH ipa a cleanruv ldif file with ALL ghost rids
> inside (really ALL from all ipa replicas, we has had some rids only
> on some replicas...)
> Example:
>
> dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task:CLEANRUV11
>
> dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task:CLEANRUV22
>
> dn: cn=replica,cn=dc\3Dexample,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task:CLEANRUV37
> ...
>
> 3. do a "ldapmodify -h 127.0.0.1 -D "cn=Directory Manager" -W -x -f
> $your-cleanruv-file.ldif" on all replicas AT THE SAME TIME :) we
> used terminator for it (https://launchpad.net/terminator). You can
> open multiple shell windows inside one window and send to all at the
> same time the same commands...
>
> 4. we have done a re-initialize of each IPA from our first master
>
> 5. restart of all replicas
>
> we are not sure about the point 3 and 4. Maybe they are not
> necessary, but we have done it.
>
> If something fails look at defect LDAP entries in whole ldap, we
> have had some entries with 'nsunique-$HASH' after the 'normal' name.
> We have deleted them.
>
> MfG
> Christoph Kaminski
>
>
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project
