Bruce Schuchardt created GEODE-6522:
---------------------------------------

             Summary: If a member is deposed as membership coordinator it never 
becomes one again
                 Key: GEODE-6522
                 URL: https://issues.apache.org/jira/browse/GEODE-6522
             Project: Geode
          Issue Type: Bug
          Components: membership
            Reporter: Bruce Schuchardt


The fix for GEODE-870 introduced a field in GmsJoinLeave that causes the view 
creator to shut down if another member becomes coordinator.  Unfortunately this 
new field is never reset after it's used, preventing the deposed node from ever 
becoming (and staying) coordinator.  This causes a lot of churn in the node and 
can cause it to fail to be able to shut down.
{noformat}
[info 2019/03/13 08:38:24.959 PDT <Geode Membership View Creator> tid=0x147] 
finished waiting for responses to view preparation

[info 2019/03/13 08:38:24.959 PDT <Geode Membership View Creator> tid=0x147] 
received new view: 
View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|15] members: 
[turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000, 
turtle(locatorgemfire_2_4_host1_17715:17715:locator)<ec><v1>:41003, 
turtle(locatorgemfire_2_2_host1_17676:17676:locator)<ec><v1>:41005]
old view is: 
View[turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000|3] 
members: [turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000, 
turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(locatorgemfire_2_4_host1_17715:17715:locator)<ec><v1>:41003, 
turtle(locatorgemfire_2_2_host1_17676:17676:locator)<ec><v1>:41005, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023]

[info 2019/03/13 08:38:24.974 PDT <Geode Membership View Creator> tid=0x147] 
Failure detection is now watching 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015; suspects are 
{turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002=View[turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000|3]
 members: [turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000, 
turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(locatorgemfire_2_4_host1_17715:17715:locator)<ec><v1>:41003, 
turtle(locatorgemfire_2_2_host1_17676:17676:locator)<ec><v1>:41005, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023]}

[info 2019/03/13 08:38:24.981 PDT <Geode Membership View Creator> tid=0x147] 
sending new view 
View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|15] members: 
[turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000, 
turtle(locatorgemfire_2_4_host1_17715:17715:locator)<ec><v1>:41003, 
turtle(locatorgemfire_2_2_host1_17676:17676:locator)<ec><v1>:41005]

[info 2019/03/13 08:38:24.981 PDT <Geode Membership View Creator> tid=0x147] 
BRUCE: setting shutdown flag in view creator
java.lang.Exception: stack trace
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.setShutdownFlag(GMSJoinLeave.java:2247)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.prepareAndSendView(GMSJoinLeave.java:2713)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.sendInitialView(GMSJoinLeave.java:2220)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.run(GMSJoinLeave.java:2299)

[info 2019/03/13 08:38:24.982 PDT <Geode Membership View Creator> tid=0x147] 
View Creator thread is exiting

[info 2019/03/13 08:38:24.982 PDT <Geode Membership View Creator> tid=0x147] 
BRUCE: setting shutdown flag in view creator
java.lang.Exception: stack trace
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.setShutdownFlag(GMSJoinLeave.java:2247)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.run(GMSJoinLeave.java:2379)

[info 2019/03/13 08:38:26.416 PDT <vm_4_thr_8_bridge_2_1_host1_17023> 
tid=0x150] GemFireCache[id = 66144348; isClosing = true; isShutDownAll = false; 
created = Wed Mar 13 08:34:41 PDT 2019; server = false; copyOnRead = false; 
lockLease = 120; lockTimeout = 60]: Now closing.



...

[info 2019/03/13 08:38:27.495 PDT <Geode Membership View Creator> tid=0x161] 
View Creator thread is starting

[info 2019/03/13 08:38:27.507 PDT <Geode Membership View Creator> tid=0x161] 
preparing new view 
View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|21] members: 
[turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019]

[info 2019/03/13 08:38:27.508 PDT <Geode Membership View Creator> tid=0x161] 
finished waiting for responses to view preparation

[info 2019/03/13 08:38:27.508 PDT <Geode Membership View Creator> tid=0x161] 
received new view: 
View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|21] members: 
[turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019]
old view is: View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|15] 
members: [turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(locatorgemfire_2_3_host1_17698:17698:locator)<ec><v0>:41000, 
turtle(locatorgemfire_2_4_host1_17715:17715:locator)<ec><v1>:41003, 
turtle(locatorgemfire_2_2_host1_17676:17676:locator)<ec><v1>:41005]

[info 2019/03/13 08:38:27.566 PDT <Geode Membership View Creator> tid=0x161] 
sending new view 
View[turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014|21] members: 
[turtle(locatorgemfire_2_1_host1_17653:17653:locator)<ec><v1>:41002, 
turtle(bridgegemfire_2_1_host1_17023:17023)<ec><v2>:41014{lead}, 
turtle(bridgegemfire_2_4_host1_17100:17100)<ec><v2>:41015, 
turtle(bridgegemfire_2_3_host1_17064:17064)<ec><v3>:41023] shutdown: 
[turtle(bridgegemfire_2_2_host1_17048:17048)<ec><v3>:41019]

[info 2019/03/13 08:38:27.567 PDT <Geode Membership View Creator> tid=0x161] 
BRUCE: setting shutdown flag in view creator
java.lang.Exception: stack trace
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.setShutdownFlag(GMSJoinLeave.java:2247)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.prepareAndSendView(GMSJoinLeave.java:2713)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.sendInitialView(GMSJoinLeave.java:2220)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$ViewCreator.run(GMSJoinLeave.java:2299)

[info 2019/03/13 08:38:27.567 PDT <Geode Membership View Creator> tid=0x161] 
View Creator thread is exiting

{noformat}
etc.

The field should be reset to false after it's used so that the next View 
Creator thread won't exit after sending its first view.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to