Sorry, I forgot the final part of the story... On Mon, Jan 26, 2009 at 09:19, Andrew Beekhof <[email protected]> wrote: > I'm seeing the following behavior somewhat regularly in my (whitetank) > cluster > > At the beginning of the test, all nodes are up and in the same cluster > partition. > We then create three partitions using iptables: > Jan 26 08:01:51 c001n16 CTS: debug: Partition[1]: ['c001n02', > 'c001n04'] > Jan 26 08:01:51 c001n16 CTS: debug: Partition[2]: ['c001n01'] > Jan 26 08:01:51 c001n16 CTS: debug: Partition[3]: ['c001n03'] > > Node uname/id table: > c001n02 156 > c001n04 158 > c001n01 155 > c001n03 157 > > This works fine. The problem occurs when we try to heal them. > We remove the iptables rules (I verified the rules were removed) and all the > nodes reform into the same partition. > > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] CLM CONFIGURATION CHANGE > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] New Configuration: > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.155) > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.156) > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.157) > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.158) > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] Members Left: > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] Members Joined: > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.155) > Jan 26 08:00:58 c001n04 openais[10345]: [CLM ] r(0) > ip(10.10.222.157) > > However cluster messaging is permanently non-functional (I've waited hours > and it doesn't resolve itself).
One can force it to resolve by picking a node and either just killing aisexec or killing _and_ restarting aisexec. I've not really noticed any pattern, but if killing aisexec isn't enough, then starting it again always does the trick. > I've attached the openais logs in the hope that someone can figure out why. And now attached is the extra logs that resulted from me killing and restarting aisexec on c001n01.
split-brain-2.log
Description: Binary data
_______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
