Make sure first you stop all nodes in one segment and only then start them, rolling restart might not fix cluster segmentation.
пт, 11 сент. 2020 г. в 09:08, Denis Magda <[email protected]>: > Hi Samuel, > > With the current behavior, the segments will not rejoin automatically. > Once the network is recovered from a network partitioning event, you need > to restart all the nodes of one of the segments. Those nodes will join the > other nodes and the cluster will become fully operational. > > Let me know if you have any other questions or guidance with this. > > - > Denis > > > On Fri, Sep 11, 2020 at 7:38 AM Samuel Ueltschi < > [email protected]> wrote: > >> Hi >> >> >> >> I've been testing Ignite (2.8.1) and it's behaviour under network >> segmentation. >> >> According to the docs, Ignite nodes should be able to detect network >> segmentation and apply the configured SegmentationPolicy. >> >> >> >> However the segmentation handling didn't trigger as I would have expected >> it to do. >> >> For my tests, I setup three cluster nodes c1, c2 and c3 running in docker >> containers, all competing for a shared IgniteLock instance in a loop. >> >> Then I used iptables in container c2 to drop all incoming and outgoing >> packages on that node. >> >> After a few seconds I got the following events: >> >> >> >> c1: >> >> - EVT_NODE_FAILED for c2 >> >> >> >> c2: >> >> - EVT_NODE_FAILED for c1 >> >> - EVT_NODE_FAILED for c3 >> >> >> >> c3: >> >> - EVT_NODE_FAILED for c2 >> >> >> >> Then I reset the iptables rules expecting that c2 would rejoin the >> cluster and detect segmentation. >> >> However this didn't happen, c2 just keeps running as a second standalone >> cluster instance. >> >> Only after restarting c2 it rejoined the cluster. >> >> >> >> Eventually I was able to trigger the EVT_NODE_SEGMENTED event by pausing >> the c2 container for 1minute. After resuming, c2 detects the segmentation >> and runs the segmentation policy as excepcted. >> >> >> >> Is this behaviour correct? Shouldn't the Ignite cluster be able to >> recover from the first scenario? >> >> During a network segmentation no packages would be able to move between >> nodes, so the iptables approach should be realistic in my oppinion. >> >> >> >> Maybe I have some wrong assumptions about network segmentation so any >> feedback would be greatly appreciated. >> >> >> >> Cheers Sam >> >> >> >> -- >> Software Engineer >> BSI Business Systems Integration AG >> Erlachstrasse 16B, CH-3012 Bern >> Telefon +41 31 850 12 06 >> >> www.bsi-software.com >> >> >> >
