On Wed, 2015-06-24 at 11:25 +0200, Ludwig Krispenz wrote:
> Oleg,
> 
> the topology plugin relies on existing connection between servers which 
> remain in a topolgy. If you remove a central node in your topology you 
> are asking for trouble.
> With Petr's patch it warns you that your topology will be disconnected, 
> and if you insist we cannot guarantee anything.
> should we completely prohibit this ? 

No, but a --force should be needed.
Without a --force option we should not allow to remove a replica
completely from another one.

> I don't know, I think you could 
> also enforce an uninstall of vm175 with probably the same result.
> what you mean be calculating the remaining topology and send it to the 
> remaining servers does not work, it would require to send a removal of a 
> segment, which would be rejected.

You would have to connect to each replica that has a replication
agreement with vm175 and remove the segment from that replica. But it
wouldn't really help much as once a replica is isolated from the central
one, it will not see the other operations going on in other replicas.

Once we have a topology resolver we will be able to warn that removing a
specific replica will cause a split brain and make very loud warnings
and even offer solutions on how to reconnect the remaining replicas, but
nothing else can really be done if the admin insist in break the
replication topology, I guess.

> The topology is broken, and I don't know how much we should invest in 
> making this info consistent on all servers.

We just need to make it very clear to the admin that replication is
broken, later on we'll have visual tools to make it easier to understand
what is going on, but that's all we can do.

> More interesting would be if we can heal this later by adding new segments.

Indeed, reconnecting all the severed replicas should cause all the
removals (segments or servers) to be replicated among servers and should
bring back the topology view in a consistent state. But not until all
servers are reconnected and replication has started again.

Simo.


> Ludwig
> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
> > Hi everybody,
> >
> > Current implementation of topology plugin (including patch 878 from 
> > Petr) allows the deletion of the central node in the star topology.
> > I had the following topology:
> >
> > vm056      vm036
> >          \         /     |
> >          vm175     |
> >          /         \     |
> > vm127       vm244
> >
> > I was able to remove node vm175 from node vm244:
> >
> > [17:54:48]ofayans@vm-244:~]$ ipa-replica-manage del 
> > vm-175.idm.lab.eng.brq.redhat.com
> > Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be 
> > disconnected:
> > Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers: 
> > vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
> > Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers: 
> > vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com, 
> > vm-127.idm.lab.eng.brq.redhat.com
> > Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers: 
> > vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com, 
> > vm-036.idm.lab.eng.brq.redhat.com
> > Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers: 
> > vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
> > Continue to delete? [no]: yes
> > Waiting for removal of replication agreements
> > unexpected error: limits exceeded for this query
> >
> > I would expect this operation to delete 4 replication agreements on 
> > all nodes:
> > vm056 - vm175
> > vm127 - vm175
> > vm244 - vm175
> > vm036 - vm175
> >
> > However an arbitrary set of replication agreements was deleted on each 
> > node leading to total infrastructure inconsistency:
> > ===============================================================
> > vm056**thought the topology was as follows:
> > vm056      vm036
> >                    /     |
> >          vm175     |
> >          /         \     |
> > vm127       vm244
> > [10:28:55]ofayans@vm-056:~]$ ipa topologysegment-find realm
> > ------------------
> > 4 segments matched
> > ------------------
> >   Segment name: 036-to-244
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-127.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> >   Left node: vm-175.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > ----------------------------
> > Number of entries returned 4
> > ----------------------------
> > ===============================================================
> > both vm036**vm244 thought the topology was as follows:
> > vm056      vm036
> >          \               |
> >          vm175     |
> >          /               |
> > vm127       vm244
> >
> > [10:26:23]ofayans@vm-036:~]$ ipa topologysegment-find
> > Suffix name: realm
> > ------------------
> > 3 segments matched
> > ------------------
> >   Segment name: 036-to-244
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-056.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-127.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > ----------------------------
> > Number of entries returned 3
> > ----------------------------
> >
> > ===============================================================
> > **vm127 thought the topology was as follows:
> > vm056      vm036
> >          \        /      |
> >          vm175     |
> >                   \      |
> > vm127       vm244
> >
> > [10:31:08]ofayans@vm-127:~]$ ipa topologysegment-find realm
> > ------------------
> > 4 segments matched
> > ------------------
> >   Segment name: 036-to-244
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-056.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> >   Left node: vm-175.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > ----------------------------
> > Number of entries returned 4
> > ----------------------------
> >
> > If I, for example, add a segment connecting vm127 and vm244, these two 
> > nodes will not synchronize the topology info:
> >
> > [10:51:03]ofayans@vm-127:~]$ ipa topologysegment-add realm 127-to-244 
> > --leftnode=vm-127.idm.lab.eng.brq.redhat.com 
> > --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
> > --------------------------
> > Added segment "127-to-244"
> > --------------------------
> >   Segment name: 127-to-244
> >   Left node: vm-127.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > [10:53:33]ofayans@vm-127:~]$ ipa topologysegment-find realm
> > ------------------
> > 5 segments matched
> > ------------------
> >   Segment name: 036-to-244
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 127-to-244
> >   Left node: vm-127.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-056.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> >   Left node: vm-175.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > ----------------------------
> > Number of entries returned 5
> > ----------------------------
> > [10:54:02]ofayans@vm-127:~]$
> >
> > =============================================================
> >
> > [10:49:38]ofayans@vm-244:~]$ ipa topologysegment-find realm
> > ------------------
> > 3 segments matched
> > ------------------
> >   Segment name: 036-to-244
> >   Left node: vm-036.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 127-to-244
> >   Left node: vm-127.idm.lab.eng.brq.redhat.com
> >   Right node: vm-244.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> >
> >   Segment name: 
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> >   Left node: vm-056.idm.lab.eng.brq.redhat.com
> >   Right node: vm-175.idm.lab.eng.brq.redhat.com
> >   Connectivity: both
> > ----------------------------
> > Number of entries returned 3
> > ----------------------------
> > [10:56:34]ofayans@vm-244:~]$
> >
> > Conclusion:
> > We either should completely prohibit the removal of the middle nodes 
> > (I mean, nodes that hide another active nodes),
> > or at the removal stage first recalculate the resulting topology and 
> > send it to all nodes before actual removal.
> > -- 
> > Oleg Fayans
> > Quality Engineer
> > FreeIPA team
> > RedHat.
> >
> >
> 


-- 
Simo Sorce * Red Hat, Inc * New York

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Reply via email to