Re: [Freeipa-devel] Topology: Central node removal in star topology

Wed, 24 Jun 2015 02:37:40 -0700 


On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:

Oleg,


the topology plugin relies on existing connection between servers 
which remain in a topolgy. If you remove a central node in your 
topology you are asking for trouble.
With Petr's patch it warns you that your topology will be 
disconnected, and if you insist we cannot guarantee anything.

Agree. I just wanted to try edge cases to see how one can break the 
system :)

should we completely prohibit this ? I don't know, I think you could 
also enforce an uninstall of vm175 with probably the same result.
what you mean be calculating the remaining topology and send it to the 
remaining servers does not work, it would require to send a removal of 
a segment, which would be rejected.



The topology is broken, and I don't know how much we should invest in 
making this info consistent on all servers.



More interesting would be if we can heal this later by adding new 
segments.

Yes, here comes the biggest question raised from this case: obviously, 
when none of the nodes possess the correct topology information 
(including the one which deleted the central node), there is no way to 
fix it by adding segments connecting the nodes that became disconnected. 
I still think that the recalculation of the resulting tree should be 
done at least on the node that performs the removal action. And when 
later some other node gets connected, it should understand somehow that 
it's topology information is outdated


Ludwig
On 06/24/2015 11:04 AM, Oleg Fayans wrote:

Hi everybody,


Current implementation of topology plugin (including patch 878 from 
Petr) allows the deletion of the central node in the star topology.

I had the following topology:

vm056      vm036
         \         /     |
         vm175     |
         /         \     |
vm127       vm244

I was able to remove node vm175 from node vm244:


[17:54:48]ofayans@vm-244:~]$ ipa-replica-manage del 
vm-175.idm.lab.eng.brq.redhat.com
Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be 
disconnected:
Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com, 
vm-127.idm.lab.eng.brq.redhat.com
Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com, 
vm-036.idm.lab.eng.brq.redhat.com
Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com

Continue to delete? [no]: yes
Waiting for removal of replication agreements
unexpected error: limits exceeded for this query


I would expect this operation to delete 4 replication agreements on 
all nodes:

vm056 - vm175
vm127 - vm175
vm244 - vm175
vm036 - vm175


However an arbitrary set of replication agreements was deleted on 
each node leading to total infrastructure inconsistency:

===============================================================
vm056**thought the topology was as follows:
vm056      vm036
                   /     |
         vm175     |
         /         \     |
vm127       vm244
[10:28:55]ofayans@vm-056:~]$ ipa topologysegment-find realm
------------------
4 segments matched
------------------
  Segment name: 036-to-244
  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-127.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com

  Left node: vm-175.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both
----------------------------
Number of entries returned 4
----------------------------
===============================================================
both vm036**vm244 thought the topology was as follows:
vm056      vm036
         \               |
         vm175     |
         /               |
vm127       vm244

[10:26:23]ofayans@vm-036:~]$ ipa topologysegment-find
Suffix name: realm
------------------
3 segments matched
------------------
  Segment name: 036-to-244
  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-056.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-127.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both
----------------------------
Number of entries returned 3
----------------------------

===============================================================
**vm127 thought the topology was as follows:
vm056      vm036
         \        /      |
         vm175     |
                  \      |
vm127       vm244

[10:31:08]ofayans@vm-127:~]$ ipa topologysegment-find realm
------------------
4 segments matched
------------------
  Segment name: 036-to-244
  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-056.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com

  Left node: vm-175.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both
----------------------------
Number of entries returned 4
----------------------------


If I, for example, add a segment connecting vm127 and vm244, these 
two nodes will not synchronize the topology info:



[10:51:03]ofayans@vm-127:~]$ ipa topologysegment-add realm 127-to-244 
--leftnode=vm-127.idm.lab.eng.brq.redhat.com 
--rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both

--------------------------
Added segment "127-to-244"
--------------------------
  Segment name: 127-to-244
  Left node: vm-127.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both
[10:53:33]ofayans@vm-127:~]$ ipa topologysegment-find realm
------------------
5 segments matched
------------------
  Segment name: 036-to-244
  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both

  Segment name: 127-to-244
  Left node: vm-127.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-056.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com

  Left node: vm-175.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both
----------------------------
Number of entries returned 5
----------------------------
[10:54:02]ofayans@vm-127:~]$

=============================================================

[10:49:38]ofayans@vm-244:~]$ ipa topologysegment-find realm
------------------
3 segments matched
------------------
  Segment name: 036-to-244
  Left node: vm-036.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both

  Segment name: 127-to-244
  Left node: vm-127.idm.lab.eng.brq.redhat.com
  Right node: vm-244.idm.lab.eng.brq.redhat.com
  Connectivity: both


  Segment name: 
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com

  Left node: vm-056.idm.lab.eng.brq.redhat.com
  Right node: vm-175.idm.lab.eng.brq.redhat.com
  Connectivity: both
----------------------------
Number of entries returned 3
----------------------------
[10:56:34]ofayans@vm-244:~]$

Conclusion:

We either should completely prohibit the removal of the middle nodes 
(I mean, nodes that hide another active nodes),
or at the removal stage first recalculate the resulting topology and 
send it to all nodes before actual removal.

--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.








--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.


-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code





















					Reply via email to