On 06/24/2015 09:01 PM, Simo Sorce wrote:
On Wed, 2015-06-24 at 11:25 +0200, Ludwig Krispenz wrote:
Oleg,

the topology plugin relies on existing connection between servers which
remain in a topolgy. If you remove a central node in your topology you
are asking for trouble.
With Petr's patch it warns you that your topology will be disconnected,
and if you insist we cannot guarantee anything.
should we completely prohibit this ?
No, but a --force should be needed.
Without a --force option we should not allow to remove a replica
completely from another one.

I don't know, I think you could
also enforce an uninstall of vm175 with probably the same result.
what you mean be calculating the remaining topology and send it to the
remaining servers does not work, it would require to send a removal of a
segment, which would be rejected.
You would have to connect to each replica that has a replication
agreement with vm175 and remove the segment from that replica. But it
wouldn't really help much as once a replica is isolated from the central
one, it will not see the other operations going on in other replicas.

Once we have a topology resolver we will be able to warn that removing a
specific replica will cause a split brain and make very loud warnings
we have this already, see the output of Oleg's example:

ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com
Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be disconnected: Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers: vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers: vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers: vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers: vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Continue to delete? [no]: yes

it tells you that the topology gets disconnected and which connections will be missing, the continue yes/no is the --force,
the question was, should we allow a force in this situation ?

More interesting would be if we can heal this later by adding new segments.
Indeed, reconnecting all the severed replicas should cause all the
removals (segments or servers) to be replicated among servers and should
bring back the topology view in a consistent state. But not until all
servers are reconnected and replication has started again.
This healing can also be required without forcing removal by an admin. If you have a start topology and your central node goes down and is not recoverable

Simo.


Ludwig
On 06/24/2015 11:04 AM, Oleg Fayans wrote:
Hi everybody,

Current implementation of topology plugin (including patch 878 from
Petr) allows the deletion of the central node in the star topology.
I had the following topology:

vm056      vm036
          \         /     |
          vm175     |
          /         \     |
vm127       vm244

I was able to remove node vm175 from node vm244:

[17:54:48]ofayans@vm-244:~]$ ipa-replica-manage del
vm-175.idm.lab.eng.brq.redhat.com
Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be
disconnected:
Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com,
vm-127.idm.lab.eng.brq.redhat.com
Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com,
vm-036.idm.lab.eng.brq.redhat.com
Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Continue to delete? [no]: yes
Waiting for removal of replication agreements
unexpected error: limits exceeded for this query

I would expect this operation to delete 4 replication agreements on
all nodes:
vm056 - vm175
vm127 - vm175
vm244 - vm175
vm036 - vm175

However an arbitrary set of replication agreements was deleted on each
node leading to total infrastructure inconsistency:
===============================================================
vm056**thought the topology was as follows:
vm056      vm036
                    /     |
          vm175     |
          /         \     |
vm127       vm244
[10:28:55]ofayans@vm-056:~]$ ipa topologysegment-find realm
------------------
4 segments matched
------------------
   Segment name: 036-to-244
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-127.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
   Left node: vm-175.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both
----------------------------
Number of entries returned 4
----------------------------
===============================================================
both vm036**vm244 thought the topology was as follows:
vm056      vm036
          \               |
          vm175     |
          /               |
vm127       vm244

[10:26:23]ofayans@vm-036:~]$ ipa topologysegment-find
Suffix name: realm
------------------
3 segments matched
------------------
   Segment name: 036-to-244
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-056.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-127.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both
----------------------------
Number of entries returned 3
----------------------------

===============================================================
**vm127 thought the topology was as follows:
vm056      vm036
          \        /      |
          vm175     |
                   \      |
vm127       vm244

[10:31:08]ofayans@vm-127:~]$ ipa topologysegment-find realm
------------------
4 segments matched
------------------
   Segment name: 036-to-244
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-056.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
   Left node: vm-175.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both
----------------------------
Number of entries returned 4
----------------------------

If I, for example, add a segment connecting vm127 and vm244, these two
nodes will not synchronize the topology info:

[10:51:03]ofayans@vm-127:~]$ ipa topologysegment-add realm 127-to-244
--leftnode=vm-127.idm.lab.eng.brq.redhat.com
--rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
--------------------------
Added segment "127-to-244"
--------------------------
   Segment name: 127-to-244
   Left node: vm-127.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both
[10:53:33]ofayans@vm-127:~]$ ipa topologysegment-find realm
------------------
5 segments matched
------------------
   Segment name: 036-to-244
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name: 127-to-244
   Left node: vm-127.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-056.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
   Left node: vm-175.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both
----------------------------
Number of entries returned 5
----------------------------
[10:54:02]ofayans@vm-127:~]$

=============================================================

[10:49:38]ofayans@vm-244:~]$ ipa topologysegment-find realm
------------------
3 segments matched
------------------
   Segment name: 036-to-244
   Left node: vm-036.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name: 127-to-244
   Left node: vm-127.idm.lab.eng.brq.redhat.com
   Right node: vm-244.idm.lab.eng.brq.redhat.com
   Connectivity: both

   Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
   Left node: vm-056.idm.lab.eng.brq.redhat.com
   Right node: vm-175.idm.lab.eng.brq.redhat.com
   Connectivity: both
----------------------------
Number of entries returned 3
----------------------------
[10:56:34]ofayans@vm-244:~]$

Conclusion:
We either should completely prohibit the removal of the middle nodes
(I mean, nodes that hide another active nodes),
or at the removal stage first recalculate the resulting topology and
send it to all nodes before actual removal.
--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.




--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to