[Freeipa-users] Re: Removing dead servers with tombstone entries

2023-06-26 Thread Joe Rhodes via FreeIPA-users 

> On Jun 23, 2023, at 08:30, Florence Blanc-Renaud  wrote:
> 
> Hi,
> 
> On Thu, Jun 22, 2023 at 3:18 PM Joe Rhodes via FreeIPA-users 
>  > wrote:
>> 
>> 
>>> On Jun 21, 2023, at 18:07, Rob Crittenden >> > wrote:
>>> 
>>> Joe Rhodes via FreeIPA-users wrote:
 Hello all!
 
 I have a CentOS 7 based FreeIPA system that I’m migrating to Rocky 9.
  As suggested, I’ve created a Rocky 8 instance replica first.
 
 As I’ve been working on this (in a dev environment first), I’ve gotten
 myself into a state where I have two servers in the config that I cannot
 delete.  (The VMs have been uninstalled and deleted.)
 
 ipa server-find
 
 -
 
 7 IPA servers matched
 
 -
 
   Server name: ia-ipa-1.dev.purestake.tech
 
   Min domain level: 0
 
   Max domain level: 1
 
 
   Server name: ia-ipa-2.dev.purestake.tech
 
   Min domain level: 0
 
   Max domain level: 1
 
 
   Server name: joe-rocky-8.dev.purestake.tech
 
   Min domain level: 1
 
   Max domain level: 1
 
 
   Server name: joe-rocky-9.dev.purestake.tech
 
   Min domain level: 1
 
   Max domain level: 1
 
 
   Server name: oh-ipa-1.dev.purestake.tech
 
   Min domain level: 0
 
   Max domain level: 1
 
 
   Server name: oh-ipa-2.dev.purestake.tech
 
   Min domain level: 0
 
   Max domain level: 1
 
 
   Server name: oh-ipa-21.dev.purestake.tech
 
   Min domain level: 1
 
   Max domain level: 1
 
 
 
 The two servers I want to delete are  joe-rocky-9  and oh-ipa-21.
 
 Trying to delete either give me:
 
 ipa server-del joe-rocky-9.dev.purestake.tech
 
 Removing joe-rocky-9.dev.purestake.tech from replication topology,
 please wait...
 
 ipa: ERROR: Server removal aborted: 
 
 
 Replication topology in suffix 'domain' is disconnected:
 
 Topology does not allow server ia-ipa-1.dev.purestake.tech to replicate
 with servers:
 
 joe-rocky-9.dev.purestake.tech
 
 Topology does not allow server ia-ipa-2.dev.purestake.tech to replicate
 with servers:
 
 joe-rocky-9.dev.purestake.tech
 
 Topology does not allow server joe-rocky-8.dev.purestake.tech to
 replicate with servers:
 
 joe-rocky-9.dev.purestake.tech
 
 Topology does not allow server joe-rocky-9.dev.purestake.tech to
 replicate with servers:
 
 joe-rocky-8.dev.purestake.tech
 
 oh-ipa-1.dev.purestake.tech
 
 oh-ipa-2.dev.purestake.tech
 
 ia-ipa-1.dev.purestake.tech
 
 oh-ipa-21.dev.purestake.tech
 
 ia-ipa-2.dev.purestake.tech
 
 Topology does not allow server oh-ipa-1.dev.purestake.tech to replicate
 with servers:
 
 joe-rocky-9.dev.purestake.tech
 
 Topology does not allow server oh-ipa-2.dev.purestake.tech to replicate
 with servers:
 
 joe-rocky-9.dev.purestake.tech
 
 Topology does not allow server oh-ipa-21.dev.purestake.tech to replicate
 with servers:
 
 joe-rocky-9.dev.purestake.tech.
 
 
 and attempting to delete, ignoring the replication topology:
 
 ipa server-del joe-rocky-9.dev.purestake.tech --ignore-topology-disconnect
 
 Removing joe-rocky-9.dev.purestake.tech from replication topology,
 please wait...
 
 ipa: ERROR: Not allowed on non-leaf entry
> This error shows that there are child entries below the entry for the server. 
> You mentioned replication conflicts, what is the output of:
> # ldapsearch -D "cn=Directory Manager" -W -b $BASEDN 
> "(&(objectClass=ldapSubEntry)(nsds5ReplConflict=*))" \* nsds5ReplConflict
> (replace $BASEDN with your base dn).
> 
> You may have to manually remove the replication conflict entries before the 
> server entry can be deleted.
> flo
> 


Flo:

YES!  This was the ldap search I needed!   
"(&(objectClass=ldapSubEntry)(nsds5ReplConflict=*))”

Once I did that, I found all my conflict entries.  I think I was missing the  
"objectClass=ldapSubEntry” in earlier searches.Your search showed me my 
conflict entries for the two servers I was trying to delete:

# oh-ipa-21.dev.purestake.tech + 33c7e594-0c6611ee-ab65dcc1-bdea5cb1, masters, 
ipa, etc, dev.purestake.tech
dn: 
cn=oh-ipa-21.dev.purestake.tech,cn=masters,cn=ipa,cn=etc,dc=dev,dc=purestake,dc=tech

# KDC + 33c7e59a-0c6611ee-ab65dcc1-bdea5cb1, oh-ipa-21.dev.purestake.tech, 
masters, ipa, etc, dev.purestake.tech
dn: 
cn=KDC+nsuniqueid=33c7e59a-0c6611ee-ab65dcc1-bdea5cb1,cn=oh-ipa-21.dev.purestake.tech,cn=masters,cn=ipa,cn=etc,dc=dev,dc=purestake,dc=tech

# oh-ipa-21.dev.purestake.tech + ea2fc894-0c6e11ee-a26cd21b-447b37f1, m

[Freeipa-users] Re: Removing dead servers with tombstone entries

2023-06-23 Thread Florence Blanc-Renaud via FreeIPA-users 
Hi,

On Thu, Jun 22, 2023 at 3:18 PM Joe Rhodes via FreeIPA-users <
freeipa-users@lists.fedorahosted.org> wrote:

>
>
> On Jun 21, 2023, at 18:07, Rob Crittenden  wrote:
>
> Joe Rhodes via FreeIPA-users wrote:
>
> Hello all!
>
> I have a CentOS 7 based FreeIPA system that I’m migrating to Rocky 9.
>  As suggested, I’ve created a Rocky 8 instance replica first.
>
> As I’ve been working on this (in a dev environment first), I’ve gotten
> myself into a state where I have two servers in the config that I cannot
> delete.  (The VMs have been uninstalled and deleted.)
>
> ipa server-find
>
> -
>
> 7 IPA servers matched
>
> -
>
>   Server name: ia-ipa-1.dev.purestake.tech
>
>   Min domain level: 0
>
>   Max domain level: 1
>
>
>   Server name: ia-ipa-2.dev.purestake.tech
>
>   Min domain level: 0
>
>   Max domain level: 1
>
>
>   Server name: joe-rocky-8.dev.purestake.tech
>
>   Min domain level: 1
>
>   Max domain level: 1
>
>
>   Server name: joe-rocky-9.dev.purestake.tech
>
>   Min domain level: 1
>
>   Max domain level: 1
>
>
>   Server name: oh-ipa-1.dev.purestake.tech
>
>   Min domain level: 0
>
>   Max domain level: 1
>
>
>   Server name: oh-ipa-2.dev.purestake.tech
>
>   Min domain level: 0
>
>   Max domain level: 1
>
>
>   Server name: oh-ipa-21.dev.purestake.tech
>
>   Min domain level: 1
>
>   Max domain level: 1
>
>
>
> The two servers I want to delete are  joe-rocky-9  and oh-ipa-21.
>
> Trying to delete either give me:
>
> ipa server-del joe-rocky-9.dev.purestake.tech
>
> Removing joe-rocky-9.dev.purestake.tech from replication topology,
> please wait...
>
> ipa: ERROR: Server removal aborted:
>
>
> Replication topology in suffix 'domain' is disconnected:
>
> Topology does not allow server ia-ipa-1.dev.purestake.tech to replicate
> with servers:
>
> joe-rocky-9.dev.purestake.tech
>
> Topology does not allow server ia-ipa-2.dev.purestake.tech to replicate
> with servers:
>
> joe-rocky-9.dev.purestake.tech
>
> Topology does not allow server joe-rocky-8.dev.purestake.tech to
> replicate with servers:
>
> joe-rocky-9.dev.purestake.tech
>
> Topology does not allow server joe-rocky-9.dev.purestake.tech to
> replicate with servers:
>
> joe-rocky-8.dev.purestake.tech
>
> oh-ipa-1.dev.purestake.tech
>
> oh-ipa-2.dev.purestake.tech
>
> ia-ipa-1.dev.purestake.tech
>
> oh-ipa-21.dev.purestake.tech
>
> ia-ipa-2.dev.purestake.tech
>
> Topology does not allow server oh-ipa-1.dev.purestake.tech to replicate
> with servers:
>
> joe-rocky-9.dev.purestake.tech
>
> Topology does not allow server oh-ipa-2.dev.purestake.tech to replicate
> with servers:
>
> joe-rocky-9.dev.purestake.tech
>
> Topology does not allow server oh-ipa-21.dev.purestake.tech to replicate
> with servers:
>
> joe-rocky-9.dev.purestake.tech.
>
>
> and attempting to delete, ignoring the replication topology:
>
> ipa server-del joe-rocky-9.dev.purestake.tech --ignore-topology-disconnect
>
> Removing joe-rocky-9.dev.purestake.tech from replication topology,
> please wait...
>
> ipa: ERROR: Not allowed on non-leaf entry
>
> This error shows that there are child entries below the entry for the
server. You mentioned replication conflicts, what is the output of:
# ldapsearch -D "cn=Directory Manager" -W -b $BASEDN
"(&(objectClass=ldapSubEntry)(nsds5ReplConflict=*))" \* nsds5ReplConflict
(replace $BASEDN with your base dn).

You may have to manually remove the replication conflict entries before the
server entry can be deleted.
flo


>
> When I do a:  ipa topologysegment-find domain the server joe-rocky-9 is
> not listed in any of the segments.
>
> I believe the issue is I have a bunch of replication issues regarding
> these two servers.  (I had been adding and removing them as I was
> finding the right way to go about my upgrade)  This command shows both
> of the servers:
>
>
> ldapsearch "nsds5ReplConflict=*"
>
>
> When I do the following search I see quite a few nsTombstone entries as
> children, which I assume is what’s blocking me from removing this DN
> (either using the ipa server-del command or the ldapdelete command).
>
>
> ldapsearch -D "cn=Directory Manager” -W "(objectclass=nsTombstone)" dn
>
>
>
> When I do this command:
>
>
> ipa-replica-manage  list-ruv
>
> Replica Update Vectors:
>
> ia-ipa-1.dev.purestake.tech:389: 4
>
> oh-ipa-1.dev.purestake.tech:389: 7
>
> ia-ipa-2.dev.purestake.tech:389: 3
>
> oh-ipa-2.dev.purestake.tech:389: 8
>
> joe-rocky-8.dev.purestake.tech:389: 19
>
> Certificate Server Replica Update Vectors:
>
> ia-ipa-1.dev.purestake.tech:389: 6
>
> joe-rocky-8.dev.purestake.tech:389: 20
>
> ia-ipa-2.dev.purestake.tech:389: 5
>
>
> I get the expected list of RUVs, without the two servers I want to
> delete.  Only the serves that are really on-line and legit show up.  So
> I cannot use the “clean-ruv” command because the bad servers don’t show
> up with a replication ID.
>
> When I do this:
>
> ipa-replica-manage -p Extraordinary-northern-Co

[Freeipa-users] Re: Removing dead servers with tombstone entries

2023-06-22 Thread Joe Rhodes via FreeIPA-users 


> On Jun 21, 2023, at 18:07, Rob Crittenden  wrote:
> 
> Joe Rhodes via FreeIPA-users wrote:
>> Hello all!
>> 
>> I have a CentOS 7 based FreeIPA system that I’m migrating to Rocky 9.
>>  As suggested, I’ve created a Rocky 8 instance replica first.
>> 
>> As I’ve been working on this (in a dev environment first), I’ve gotten
>> myself into a state where I have two servers in the config that I cannot
>> delete.  (The VMs have been uninstalled and deleted.)
>> 
>> ipa server-find
>> 
>> -
>> 
>> 7 IPA servers matched
>> 
>> -
>> 
>>   Server name: ia-ipa-1.dev.purestake.tech
>> 
>>   Min domain level: 0
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: ia-ipa-2.dev.purestake.tech
>> 
>>   Min domain level: 0
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: joe-rocky-8.dev.purestake.tech
>> 
>>   Min domain level: 1
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: joe-rocky-9.dev.purestake.tech
>> 
>>   Min domain level: 1
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: oh-ipa-1.dev.purestake.tech
>> 
>>   Min domain level: 0
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: oh-ipa-2.dev.purestake.tech
>> 
>>   Min domain level: 0
>> 
>>   Max domain level: 1
>> 
>> 
>>   Server name: oh-ipa-21.dev.purestake.tech
>> 
>>   Min domain level: 1
>> 
>>   Max domain level: 1
>> 
>> 
>> 
>> The two servers I want to delete are  joe-rocky-9  and oh-ipa-21.
>> 
>> Trying to delete either give me:
>> 
>> ipa server-del joe-rocky-9.dev.purestake.tech
>> 
>> Removing joe-rocky-9.dev.purestake.tech from replication topology,
>> please wait...
>> 
>> ipa: ERROR: Server removal aborted: 
>> 
>> 
>> Replication topology in suffix 'domain' is disconnected:
>> 
>> Topology does not allow server ia-ipa-1.dev.purestake.tech to replicate
>> with servers:
>> 
>> joe-rocky-9.dev.purestake.tech
>> 
>> Topology does not allow server ia-ipa-2.dev.purestake.tech to replicate
>> with servers:
>> 
>> joe-rocky-9.dev.purestake.tech
>> 
>> Topology does not allow server joe-rocky-8.dev.purestake.tech to
>> replicate with servers:
>> 
>> joe-rocky-9.dev.purestake.tech
>> 
>> Topology does not allow server joe-rocky-9.dev.purestake.tech to
>> replicate with servers:
>> 
>> joe-rocky-8.dev.purestake.tech
>> 
>> oh-ipa-1.dev.purestake.tech
>> 
>> oh-ipa-2.dev.purestake.tech
>> 
>> ia-ipa-1.dev.purestake.tech
>> 
>> oh-ipa-21.dev.purestake.tech
>> 
>> ia-ipa-2.dev.purestake.tech
>> 
>> Topology does not allow server oh-ipa-1.dev.purestake.tech to replicate
>> with servers:
>> 
>> joe-rocky-9.dev.purestake.tech
>> 
>> Topology does not allow server oh-ipa-2.dev.purestake.tech to replicate
>> with servers:
>> 
>> joe-rocky-9.dev.purestake.tech
>> 
>> Topology does not allow server oh-ipa-21.dev.purestake.tech to replicate
>> with servers:
>> 
>> joe-rocky-9.dev.purestake.tech.
>> 
>> 
>> and attempting to delete, ignoring the replication topology:
>> 
>> ipa server-del joe-rocky-9.dev.purestake.tech --ignore-topology-disconnect
>> 
>> Removing joe-rocky-9.dev.purestake.tech from replication topology,
>> please wait...
>> 
>> ipa: ERROR: Not allowed on non-leaf entry
>> 
>> 
>> When I do a:  ipa topologysegment-find domain the server joe-rocky-9 is
>> not listed in any of the segments.
>> 
>> I believe the issue is I have a bunch of replication issues regarding
>> these two servers.  (I had been adding and removing them as I was
>> finding the right way to go about my upgrade)  This command shows both
>> of the servers:
>> 
>> 
>> ldapsearch "nsds5ReplConflict=*"
>> 
>> 
>> When I do the following search I see quite a few nsTombstone entries as
>> children, which I assume is what’s blocking me from removing this DN
>> (either using the ipa server-del command or the ldapdelete command).
>> 
>> 
>> ldapsearch -D "cn=Directory Manager” -W "(objectclass=nsTombstone)" dn
>> 
>> 
>> 
>> When I do this command:
>> 
>> 
>> ipa-replica-manage  list-ruv
>> 
>> Replica Update Vectors:
>> 
>> ia-ipa-1.dev.purestake.tech:389: 4
>> 
>> oh-ipa-1.dev.purestake.tech:389: 7
>> 
>> ia-ipa-2.dev.purestake.tech:389: 3
>> 
>> oh-ipa-2.dev.purestake.tech:389: 8
>> 
>> joe-rocky-8.dev.purestake.tech:389: 19
>> 
>> Certificate Server Replica Update Vectors:
>> 
>> ia-ipa-1.dev.purestake.tech:389: 6
>> 
>> joe-rocky-8.dev.purestake.tech:389: 20
>> 
>> ia-ipa-2.dev.purestake.tech:389: 5
>> 
>> 
>> I get the expected list of RUVs, without the two servers I want to
>> delete.  Only the serves that are really on-line and legit show up.  So
>> I cannot use the “clean-ruv” command because the bad servers don’t show
>> up with a replication ID.
>> 
>> When I do this:
>> 
>> ipa-replica-manage -p Extraordinary-northern-Conditioning-Idaho-7
>> clean-dangling-ruv
>> 
>> 
>> The server 'joe-rocky-9.dev.purestake.tech' appears to be offline.
>> 
>> The server 'oh-ipa-21.dev.purestake.tech' appears to be offline.
>> 
>> No dangling RUVs found
>> 
>> 
>> 
>> I see the

[Freeipa-users] Re: Removing dead servers with tombstone entries

2023-06-21 Thread Rob Crittenden via FreeIPA-users 
Joe Rhodes via FreeIPA-users wrote:
> Hello all!
> 
> I have a CentOS 7 based FreeIPA system that I’m migrating to Rocky 9.
>  As suggested, I’ve created a Rocky 8 instance replica first.
> 
> As I’ve been working on this (in a dev environment first), I’ve gotten
> myself into a state where I have two servers in the config that I cannot
> delete.  (The VMs have been uninstalled and deleted.)
> 
> ipa server-find
> 
> -
> 
> 7 IPA servers matched
> 
> -
> 
>   Server name: ia-ipa-1.dev.purestake.tech
> 
>   Min domain level: 0
> 
>   Max domain level: 1
> 
> 
>   Server name: ia-ipa-2.dev.purestake.tech
> 
>   Min domain level: 0
> 
>   Max domain level: 1
> 
> 
>   Server name: joe-rocky-8.dev.purestake.tech
> 
>   Min domain level: 1
> 
>   Max domain level: 1
> 
> 
>   Server name: joe-rocky-9.dev.purestake.tech
> 
>   Min domain level: 1
> 
>   Max domain level: 1
> 
> 
>   Server name: oh-ipa-1.dev.purestake.tech
> 
>   Min domain level: 0
> 
>   Max domain level: 1
> 
> 
>   Server name: oh-ipa-2.dev.purestake.tech
> 
>   Min domain level: 0
> 
>   Max domain level: 1
> 
> 
>   Server name: oh-ipa-21.dev.purestake.tech
> 
>   Min domain level: 1
> 
>   Max domain level: 1
> 
> 
> 
> The two servers I want to delete are  joe-rocky-9  and oh-ipa-21.
> 
> Trying to delete either give me:
> 
> ipa server-del joe-rocky-9.dev.purestake.tech
> 
> Removing joe-rocky-9.dev.purestake.tech from replication topology,
> please wait...
> 
> ipa: ERROR: Server removal aborted: 
> 
> 
> Replication topology in suffix 'domain' is disconnected:
> 
> Topology does not allow server ia-ipa-1.dev.purestake.tech to replicate
> with servers:
> 
>     joe-rocky-9.dev.purestake.tech
> 
> Topology does not allow server ia-ipa-2.dev.purestake.tech to replicate
> with servers:
> 
>     joe-rocky-9.dev.purestake.tech
> 
> Topology does not allow server joe-rocky-8.dev.purestake.tech to
> replicate with servers:
> 
>     joe-rocky-9.dev.purestake.tech
> 
> Topology does not allow server joe-rocky-9.dev.purestake.tech to
> replicate with servers:
> 
>     joe-rocky-8.dev.purestake.tech
> 
>     oh-ipa-1.dev.purestake.tech
> 
>     oh-ipa-2.dev.purestake.tech
> 
>     ia-ipa-1.dev.purestake.tech
> 
>     oh-ipa-21.dev.purestake.tech
> 
>     ia-ipa-2.dev.purestake.tech
> 
> Topology does not allow server oh-ipa-1.dev.purestake.tech to replicate
> with servers:
> 
>     joe-rocky-9.dev.purestake.tech
> 
> Topology does not allow server oh-ipa-2.dev.purestake.tech to replicate
> with servers:
> 
>     joe-rocky-9.dev.purestake.tech
> 
> Topology does not allow server oh-ipa-21.dev.purestake.tech to replicate
> with servers:
> 
>     joe-rocky-9.dev.purestake.tech.
> 
> 
> and attempting to delete, ignoring the replication topology:
> 
> ipa server-del joe-rocky-9.dev.purestake.tech --ignore-topology-disconnect
> 
> Removing joe-rocky-9.dev.purestake.tech from replication topology,
> please wait...
> 
> ipa: ERROR: Not allowed on non-leaf entry
> 
> 
> When I do a:  ipa topologysegment-find domain the server joe-rocky-9 is
> not listed in any of the segments.
> 
> I believe the issue is I have a bunch of replication issues regarding
> these two servers.  (I had been adding and removing them as I was
> finding the right way to go about my upgrade)  This command shows both
> of the servers:
> 
> 
> ldapsearch "nsds5ReplConflict=*"
> 
> 
> When I do the following search I see quite a few nsTombstone entries as
> children, which I assume is what’s blocking me from removing this DN
> (either using the ipa server-del command or the ldapdelete command).
> 
> 
> ldapsearch -D "cn=Directory Manager” -W "(objectclass=nsTombstone)" dn
> 
> 
> 
> When I do this command:
> 
> 
> ipa-replica-manage  list-ruv
> 
> Replica Update Vectors:
> 
> ia-ipa-1.dev.purestake.tech:389: 4
> 
> oh-ipa-1.dev.purestake.tech:389: 7
> 
> ia-ipa-2.dev.purestake.tech:389: 3
> 
> oh-ipa-2.dev.purestake.tech:389: 8
> 
> joe-rocky-8.dev.purestake.tech:389: 19
> 
> Certificate Server Replica Update Vectors:
> 
> ia-ipa-1.dev.purestake.tech:389: 6
> 
> joe-rocky-8.dev.purestake.tech:389: 20
> 
> ia-ipa-2.dev.purestake.tech:389: 5
> 
> 
> I get the expected list of RUVs, without the two servers I want to
> delete.  Only the serves that are really on-line and legit show up.  So
> I cannot use the “clean-ruv” command because the bad servers don’t show
> up with a replication ID.
> 
> When I do this:
> 
> ipa-replica-manage -p Extraordinary-northern-Conditioning-Idaho-7
> clean-dangling-ruv
> 
> 
> The server 'joe-rocky-9.dev.purestake.tech' appears to be offline.
> 
> The server 'oh-ipa-21.dev.purestake.tech' appears to be offline.
> 
> No dangling RUVs found
> 
> 
> 
> I see the two problematic entries timing out (as expected, since they
> don’t exist).
> 
> I’m just not sure how to remove these two dead servers.  It seems like I
> need to resolve or delete the nsTombstone children, but that doesn’t
> seem to be possible.
> 
> I’m kind o