Hello Team,

I faced a strange issue today where suspected member was forcefully 
disconnected from distributed system due to lack of heartbeats.
But later on after so many attempts,


Ø  Suspected member completely disconnected [DistributionManager closed]

Ø  Attempted to reconnect to distributed system by starting membership 
services, Jgroup channel, Gemfire P2P listener

Ø  Failed due to following exception

org.apache.geode.security.AuthenticationRequiredException: Failed to find 
credentials from [host001(event-server-1:3525)<ec>:1026]]

This looks very strange or may be issue. It was part of distributed system 
using username/password and security manager from beginning. But after force 
disconnection, when it attempted to join distributed system back, it was unable 
to find credentials itself.
Could someone help me to validate it?

More,
Even though server was no more part of distributed system after all these 
events, spring boot geode app was still running.
Shouldn't that also be stopped? I had to manually kill that for rectifying this.

I have attached detailed logs as well.

Topology

Version: Geode 1.1.1
Java: JDK 8
Platform: Red Hat Enterprise Linux

Host1:
Locator1
EventServer1 [Group = Events]
HopServer1 [Group = Hops]

Host2:
Locatro2
EventServer2 [Group = Events]
HopServer2 [Group = Hops]

Thanks,
Dharam


This message is confidential and subject to terms at: 
http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal 
privilege, viruses and monitoring of electronic messages. If you are not the 
intended recipient, please delete this message and notify the sender 
immediately. Any unauthorized use is strictly prohibited.
[info 2018/04/10 02:39:19.324 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] received suspect message from 
host002(event-server-2:1487)<ec><v31>:1026 for host001(Locator1:1134
:locator)<ec><v16>:1024: Member isn't responding to heartbeat requests

[warning 2018/04/10 02:39:20.688 EDT event-server-1 <Management Task> tid=0x58] 
Attempting TCP/IP reconnect to  host002(Locator2:3605:locator)<ec><v29>:1024

[warning 2018/04/10 02:39:20.707 EDT event-server-1 <P2P message reader for 
host002(event-server-2:1487)<ec><v31>:1026 shared ordered uid=8 port=43352> 
tid=0x189] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[warning 2018/04/10 02:39:33.778 EDT event-server-1 <Cache Server Load Polling 
Thread> tid=0xf5] Attempting TCP/IP reconnect to  
host002(Locator2:3605:locator)<ec><v29>:1024

[warning 2018/04/10 02:39:33.980 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[warning 2018/04/10 02:39:35.790 EDT event-server-1 <ServerConnection on port 
40404 Thread 10> tid=0x124] 15 seconds have elapsed while waiting for replies: 
<DistributedCacheOperation$CacheOperationReplyProcessor 513164 waiting for 1 
replies from [host002(event-server-2:1487)<ec><v31>:1026]> on 
host001(event-server-1:3525)<ec><v18>:1026 whose current membership list is: 
[[host001(event-server-1:3525)<ec><v18>:1026, 
host001(Locator1:1134:locator)<ec><v16>:1024, 
host002(event-server-2:1487)<ec><v31>:1026, 
host002(hp-server-2:1418)<ec><v30>:1025, 
host002(Locator2:3605:locator)<ec><v29>:1024]]

[info 2018/04/10 02:39:36.837 EDT event-server-1 <P2P message reader for 
host002(Locator2:3605:locator)<ec><v29>:1024 shared unordered uid=1 port=42338> 
tid=0x17c] Performing final check for suspect member 
host002(Locator2:3605:locator)<ec><v29>:1024 reason=member unexpectedly shut 
down shared, unordered connection

[info 2018/04/10 02:39:36.851 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] Membership received a request to remove 
host001(Locator1:1134:locator)<ec><v16>:1024 from 
host002(Locator2:3605:locator)<ec><v29>:1024 reason=Member isn't responding to 
heartbeat requests

[info 2018/04/10 02:39:36.852 EDT event-server-1 <ServerConnection on port 
40404 Thread 2> tid=0x103] Server connection from 
[identity(host002(1634:loner):34942:6531c5a4,connection=1; port=44006]: 
connection disconnect detected by EOF.

[info 2018/04/10 02:39:37.365 EDT event-server-1 <P2P message reader for 
host002(hp-server-2:1418)<ec><v30>:1025 shared unordered uid=1 port=43112> 
tid=0x181] Performing final check for suspect member 
host002(hp-server-2:1418)<ec><v30>:1025 reason=member unexpectedly shut down 
shared, unordered connection

[info 2018/04/10 02:39:37.301 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] This member is becoming the membership 
coordinator with address host001(event-server-1:3525)<ec><v18>:1026

[info 2018/04/10 02:39:37.281 EDT event-server-1 <Cache Server Load Polling 
Thread> tid=0xf5] Successfully reconnected to member 
host002(Locator2:3605:locator)<ec><v29>:1024

[info 2018/04/10 02:39:37.276 EDT event-server-1 <Management Task> tid=0x58] 
Successfully reconnected to member host002(Locator2:3605:locator)<ec><v29>:1024

[info 2018/04/10 02:39:37.772 EDT event-server-1 <P2P message reader for 
host002(event-server-2:1487)<ec><v31>:1026 shared ordered uid=8 port=43352> 
tid=0x189] Successfully reconnected to member 
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:37.820 EDT event-server-1 <P2P message reader for 
host002(event-server-2:1487)<ec><v31>:1026 shared unordered uid=1 port=43268> 
tid=0x186] Performing final check for suspect member 
host002(event-server-2:1487)<ec><v31>:1026 reason=member unexpectedly shut down 
shared, unordered connection

[info 2018/04/10 02:39:37.821 EDT event-server-1 <Geode Failure Detection 
thread 19> tid=0x207] Performing final check for suspect member 
host001(Locator1:1134:locator)<ec><v16>:1024 reason=Member isn't responding to 
heartbeat requests

[warning 2018/04/10 02:39:37.930 EDT event-server-1 <ServerConnection on port 
40404 Thread 2> tid=0x103] ClientHealthMonitor: Unregistering client with 
member id identity(host002(1634:loner):34942:6531c5a4,connection=1 due to: The 
connection has been reset while reading the header

[info 2018/04/10 02:39:37.930 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] Successfully reconnected to member 
host002(event-server-2:1487)<ec><v31>:1026

[warning 2018/04/10 02:39:38.039 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] Attempting TCP/IP reconnect to  
host002(hp-server-2:1418)<ec><v30>:1025

[info 2018/04/10 02:39:38.725 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] Successfully reconnected to member 
host002(hp-server-2:1418)<ec><v30>:1025

[info 2018/04/10 02:39:38.769 EDT event-server-1 <Geode Failure Detection 
thread 16> tid=0x1fa] received suspect message from 
host001(event-server-1:3525)<ec><v18>:1026 for 
host002(event-server-2:1487)<ec><v31>:1026: Member isn't responding to 
heartbeat requests

[info 2018/04/10 02:39:38.830 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] Server connection from 
[identity(host001(8115:loner):50832:190583a4,connection=1; port=47752]: 
connection disconnect detected by EOF.

[warning 2018/04/10 02:39:38.831 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] ClientHealthMonitor: Unregistering client with 
member id identity(host001(8115:loner):50832:190583a4,connection=1 due to: The 
connection has been reset while reading the header

[warning 2018/04/10 02:39:39.215 EDT event-server-1 <Management Task> tid=0x58] 
Attempting TCP/IP reconnect to  host002(Locator2:3605:locator)<ec><v29>:1024

[info 2018/04/10 02:39:39.265 EDT event-server-1 <P2P message reader for 
host002(event-server-2:1487)<ec><v31>:1026 shared unordered uid=1 port=43268> 
tid=0x186] Final check passed for suspect member 
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:39.269 EDT event-server-1 <P2P message reader for 
host002(Locator2:3605:locator)<ec><v29>:1024 shared unordered uid=1 port=42338> 
tid=0x17c] Final check passed for suspect member 
host002(Locator2:3605:locator)<ec><v29>:1024

[info 2018/04/10 02:39:39.265 EDT event-server-1 <P2P message reader for 
host002(hp-server-2:1418)<ec><v30>:1025 shared unordered uid=1 port=43112> 
tid=0x181] Final check passed for suspect member 
host002(hp-server-2:1418)<ec><v30>:1025

[warning 2018/04/10 02:39:39.469 EDT event-server-1 <Cache Server Load Polling 
Thread> tid=0xf5] Attempting TCP/IP reconnect to  
host002(Locator2:3605:locator)<ec><v29>:1024

[warning 2018/04/10 02:39:39.484 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:39.484 EDT event-server-1 <Geode Failure Detection 
thread 19> tid=0x207] Final check passed for suspect member 
host001(Locator1:1134:locator)<ec><v16>:1024

[warning 2018/04/10 02:39:43.118 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[severe 2018/04/10 02:39:43.301 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] This member is no longer in the membership 
view.  My ID is host001(event-server-1:3525)<ec><v18>:1026 and the new view is 
View[host002(Locator2:3605:locator)<ec><v29>:1024|38] members: 
[host002(Locator2:3605:locator)<ec><v29>:1024, 
host002(hp-server-2:1418)<ec><v30>:1025{lead}, 
host002(event-server-2:1487)<ec><v31>:1026]  crashed: 
[host001(Locator1:1134:locator)<ec><v16>:1024, 
host001(event-server-1:3525)<ec><v18>:1026]

[info 2018/04/10 02:39:43.325 EDT event-server-1 <Geode Membership View 
Creator> tid=0x20d] View Creator thread is starting

[warning 2018/04/10 02:39:43.434 EDT event-server-1 <Management Task> tid=0x58] 
Attempting TCP/IP reconnect to  host002(Locator2:3605:locator)<ec><v29>:1024

[info 2018/04/10 02:39:43.544 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] Successfully reconnected to member 
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:43.614 EDT event-server-1 <Geode Membership View 
Creator> tid=0x20d] preparing new view 
View[host001(event-server-1:3525)<ec><v18>:1026|46] members: 
[host001(event-server-1:3525)<ec><v18>:1026{lead}, 
host002(Locator2:3605:locator)<ec><v29>:1024, 
host002(hp-server-2:1418)<ec><v30>:1025, 
host002(event-server-2:1487)<ec><v31>:1026]  crashed: 
[host001(Locator1:1134:locator)<ec><v16>:1024]
  failure detection ports: 12988 44464 27210 21079

[warning 2018/04/10 02:39:43.658 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:43.841 EDT event-server-1 <ServerConnection on port 
40404 Thread 6> tid=0x10c] Successfully reconnected to member 
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:43.993 EDT event-server-1 <Management Task> tid=0x58] 
Successfully reconnected to member host002(Locator2:3605:locator)<ec><v29>:1024

[warning 2018/04/10 02:39:43.992 EDT event-server-1 <ServerConnection on port 
40404 Thread 7> tid=0x10d] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:44.016 EDT event-server-1 <ServerConnection on port 
40404 Thread 7> tid=0x10d] Ending reconnect attempt to 
host002(event-server-2:1487)<ec><v31>:1026 because shutdown has started.

[warning 2018/04/10 02:39:44.089 EDT event-server-1 <ServerConnection on port 
40404 Thread 2> tid=0x103] Attempting TCP/IP reconnect to  
host002(event-server-2:1487)<ec><v31>:1026

[info 2018/04/10 02:39:44.089 EDT event-server-1 <ServerConnection on port 
40404 Thread 2> tid=0x103] Ending reconnect attempt to 
host002(event-server-2:1487)<ec><v31>:1026 because shutdown has started.

[info 2018/04/10 02:39:44.199 EDT event-server-1 <Queue Removal Thread> 
tid=0x102] The QueueRemovalThread is done.

[severe 2018/04/10 02:39:44.379 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] Membership service failure: This node is no 
longer in the membership view
org.apache.geode.ForcedDisconnectException: This node is no longer in the 
membership view
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2520)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:998)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processViewMessage(GMSJoinLeave.java:984)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1690)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1286)
        at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
        at org.jgroups.JChannel.up(JChannel.java:741)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
        at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
        at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
        at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
        at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
        at org.jgroups.protocols.TP.receive(TP.java:1654)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
         at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
        at java.lang.Thread.run(Thread.java:745)

[info 2018/04/10 02:39:44.683 EDT event-server-1 <unicast 
receiver,host001-11901> tid=0x33] CacheServer configuration saved

[info 2018/04/10 02:39:44.848 EDT event-server-1 <Geode Membership View 
Creator> tid=0x20d] finished waiting for responses to view preparation

[info 2018/04/10 02:39:45.815 EDT event-server-1 <DisconnectThread> tid=0x217] 
Stopping membership services

[info 2018/04/10 02:39:45.966 EDT event-server-1 <DisconnectThread> tid=0x217] 
GMSHealthMonitor server socket is closed in stopServices().

[warning 2018/04/10 02:39:46.148 EDT event-server-1 <ServerConnection on port 
40404 Thread 2> tid=0x103] Server connection from 
[identity(host001(8115:loner):50832:190583a4,connection=2; port=49136]: 
Unexpected Exception
org.apache.geode.distributed.DistributedSystemDisconnectedException: 
DistributedSystem is shutting down, caused by 
org.apache.geode.ForcedDisconnectException: This node is no longer in the 
membership view
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1700)
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
        at 
org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
        at 
org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
        at 
org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
        at 
org.apache.geode.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1832)
        at 
org.apache.geode.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:505)
        at 
org.apache.geode.internal.cache.DistributedRegion.distributeDestroy(DistributedRegion.java:1701)
        at 
org.apache.geode.internal.cache.DistributedRegion.basicDestroyPart3(DistributedRegion.java:1692)
        at 
org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1519)
        at 
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6778)
        at 
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6755)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
        at 
org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6717)
        at 
org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1662)
        at 
org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5614)
        at 
org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:239)
        at 
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
        at 
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
        at 
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
        at 
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1138)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at 
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.geode.ForcedDisconnectException: This node is no longer 
in the membership view
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2520)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:998)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processViewMessage(GMSJoinLeave.java:984)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1690)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1286)
        at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
        at org.jgroups.JChannel.up(JChannel.java:741)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
        at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
        at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
        at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
        at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
        at org.jgroups.protocols.TP.receive(TP.java:1654)
        at 
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
        at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
        ... 1 more

[info 2018/04/10 02:39:46.400 EDT event-server-1 <Geode Failure Detection 
Server thread 0> tid=0x37] GMSHealthMonitor server thread exiting

[info 2018/04/10 02:39:46.419 EDT event-server-1 <DisconnectThread> tid=0x217] 
GMSHealthMonitor serverSocketExecutor is terminated

[info 2018/04/10 02:39:48.577 EDT event-server-1 <Cache Server Selector 
/0.0.0.0:40404 local port: 40404> tid=0xfa] Cache server on port 40,404 is 
shutting down.

[info 2018/04/10 02:39:49.200 EDT event-server-1 <ReconnectThread> tid=0x217] 
Disconnecting old DistributedSystem to prepare for a reconnect attempt

[info 2018/04/10 02:39:49.757 EDT event-server-1 <ReconnectThread> tid=0x217] 
GemFireCache[id = 1710483461; isClosing = true; isShutDownAll = false; created 
= Sun Apr 08 05:04:20 EDT 2018; server = false; copyOnRead = false; lockLease = 
120; lockTimeout = 60]: Now closing.

[info 2018/04/10 02:39:56.831 EDT event-server-1 <ReconnectThread> tid=0x217] 
Created oplog#2 krf for disk store events_disk_store.

[info 2018/04/10 02:39:56.847 EDT event-server-1 <ReconnectThread> tid=0x217] 
Created oplog#2 krf for disk store pdx_metadata_diskstore.

[info 2018/04/10 02:39:58.157 EDT event-server-1 <ReconnectThread> tid=0x217] 
Rest Server on port 9,301 is shutting down

[info 2018/04/10 02:39:58.176 EDT event-server-1 <ReconnectThread> tid=0x217] 
Stopping the HTTP service...

[info 2018/04/10 02:39:59.229 EDT event-server-1 <ReconnectThread> tid=0x217] 
Shutting down DistributionManager host001(event-server-1:3525)<ec><v18>:1026. 
At least one Exception occurred.

[info 2018/04/10 02:39:59.420 EDT event-server-1 <ReconnectThread> tid=0x217] 
Now closing distribution for host001(event-server-1:3525)<ec><v18>:1026

[info 2018/04/10 02:39:59.423 EDT event-server-1 <ReconnectThread> tid=0x217] 
DistributionManager stopped in 193ms.

[info 2018/04/10 02:39:59.423 EDT event-server-1 <ReconnectThread> tid=0x217] 
Marking DistributionManager host001(event-server-1:3525)<ec><v18>:1026 as 
closed.


[info 2018/04/10 02:40:59.470 EDT event-server-1 <ReconnectThread> tid=0x217] 
Startup Configuration:
   ### GemFire Properties defined with api ###
  ack-severe-alert-threshold=0
  ack-wait-threshold=15
  archive-disk-space-limit=0
  archive-file-size-limit=0
  async-distribution-timeout=0
  async-max-queue-size=8
  async-queue-timeout=60000
  bind-address=
  cache-xml-file=cache.xml
  cluster-configuration-dir=
  cluster-ssl-ciphers=any
  cluster-ssl-enabled=false
  cluster-ssl-keystore=
  cluster-ssl-keystore-password=********
  cluster-ssl-keystore-type=
  cluster-ssl-protocols=any
  cluster-ssl-require-authentication=true
  cluster-ssl-truststore=
  cluster-ssl-truststore-password=********
  conflate-events=server
  conserve-sockets=true
  delta-propagation=true
  deploy-working-dir=/home/geode/deploy
  disable-auto-reconnect=false
  disable-tcp=false
  distributed-system-id=-1
  distributed-transactions=false
  durable-client-id=
  durable-client-timeout=300
  enable-cluster-configuration=true
  enable-network-partition-detection=false
  enable-time-statistics=false
  enforce-unique-host=false
  gateway-ssl-ciphers=any
  gateway-ssl-enabled=false
  gateway-ssl-keystore=
  gateway-ssl-keystore-password=********
  gateway-ssl-keystore-type=
  gateway-ssl-protocols=any
  gateway-ssl-require-authentication=true
  gateway-ssl-truststore=
  gateway-ssl-truststore-password=********
  groups=INSTRUMENTS
  http-service-bind-address=host001
  http-service-port=9301
  http-service-ssl-ciphers=any
  http-service-ssl-enabled=false
  http-service-ssl-keystore=
  http-service-ssl-keystore-password=********
  http-service-ssl-keystore-type=
  http-service-ssl-protocols=any
  http-service-ssl-require-authentication=false
  http-service-ssl-truststore=
  http-service-ssl-truststore-password=********
  jmx-manager=false
  jmx-manager-access-file=
  jmx-manager-bind-address=
  jmx-manager-hostname-for-clients=
  jmx-manager-http-port=9301
   jmx-manager-password-file=********
  jmx-manager-port=1099
  jmx-manager-ssl-ciphers=any
  jmx-manager-ssl-enabled=false
  jmx-manager-ssl-keystore=
  jmx-manager-ssl-keystore-password=********
  jmx-manager-ssl-keystore-type=
  jmx-manager-ssl-protocols=any
  jmx-manager-ssl-require-authentication=true
  jmx-manager-ssl-truststore=
  jmx-manager-ssl-truststore-password=********
  jmx-manager-start=false
  jmx-manager-update-rate=2000
  load-cluster-configuration-from-dir=false
  locator-wait-time=0
  locators=host001.company.com[10334],host002.company.com[10334]
  lock-memory=false
  log-disk-space-limit=2048
  log-file=/home/logs/current/event-server-1/event-server-1.log
  log-file-size-limit=100
  log-level=info
  max-num-reconnect-tries=3
  max-wait-time-reconnect=60000
  mcast-address=239.192.81.1
  mcast-flow-control=1048576, 0.25, 5000
  mcast-port=0
  mcast-recv-buffer-size=1048576
  mcast-send-buffer-size=65535
  mcast-ttl=32
  member-timeout=5000
  membership-port-range=1024-65535
  memcached-bind-address=
  memcached-port=0
  memcached-protocol=ASCII
  name=event-server-1
  off-heap-memory-size=
  redis-bind-address=
  redis-password=********
  redis-port=0
  redundancy-zone=
  remote-locators=
  remove-unresponsive-client=false
  roles=
  security-client-accessor=********
  security-client-accessor-pp=********
  security-client-auth-init=********
  security-client-authenticator=********
  security-client-dhalgo=********
  security-log-file=********
  security-log-level=********
  security-manager=********
  security-peer-auth-init=********
  security-peer-authenticator=********
  security-peer-verifymember-timeout=********
  security-post-processor=********
  security-udp-dhalgo=********
  server-bind-address=
  server-ssl-ciphers=any
  server-ssl-enabled=false
  server-ssl-keystore=
   server-ssl-keystore-password=********
  server-ssl-keystore-type=
  server-ssl-protocols=any
  server-ssl-require-authentication=true
  server-ssl-truststore=
  server-ssl-truststore-password=********
  socket-buffer-size=32768
  socket-lease-time=60000
  ssl-ciphers=any
  ssl-cluster-alias=
  ssl-default-alias=
  ssl-enabled-components=
  ssl-gateway-alias=
  ssl-jmx-alias=
  ssl-keystore=
  ssl-keystore-password=********
  ssl-keystore-type=
  ssl-locator-alias=
  ssl-protocols=any
  ssl-require-authentication=true
  ssl-server-alias=
  ssl-truststore=
  ssl-truststore-password=********
  ssl-web-alias=
  ssl-web-require-authentication=false
  start-dev-rest-api=true
  start-locator=
  statistic-archive-file=
  statistic-sample-rate=1000
  statistic-sampling-enabled=true
  tcp-port=0
  udp-fragment-size=60000
  udp-recv-buffer-size=1048576
  udp-send-buffer-size=65535
  use-cluster-configuration=true
  user-command-packages=


[info 2018/04/10 02:40:59.426 EDT event-server-1 <ReconnectThread> tid=0x217] 
Attempting to reconnect to the distributed system.  This is attempt #1.


[info 2018/04/10 02:40:59.489 EDT event-server-1 <ReconnectThread> tid=0x217] 
Starting membership services

[info 2018/04/10 02:40:59.497 EDT event-server-1 <ReconnectThread> tid=0x217] 
JGroups channel reinitialized (took 8ms)

[info 2018/04/10 02:40:59.499 EDT event-server-1 <ReconnectThread> tid=0x217] 
GemFire P2P Listener started on host001.company.com/169.87.179.47:34532

[info 2018/04/10 02:40:59.505 EDT event-server-1 <Geode Failure Detection 
Server thread 0> tid=0x232] Started failure detection server thread on 
/169.87.179.47:62294.

[info 2018/04/10 02:40:59.516 EDT event-server-1 <ReconnectThread> tid=0x217] 
Attempting to join the distributed system through coordinator 
host002(Locator2:3605:locator)<ec><v29>:1024 using address 
host001(event-server-1:3525)<ec>:1026

[info 2018/04/10 02:40:59.522 EDT event-server-1 <ReconnectThread> tid=0x217] 
Stopping membership services

[info 2018/04/10 02:40:59.534 EDT event-server-1 <ReconnectThread> tid=0x217] 
GMSHealthMonitor server socket is closed in stopServices().

[info 2018/04/10 02:40:59.534 EDT event-server-1 <Geode Failure Detection 
Server thread 0> tid=0x232] GMSHealthMonitor server thread exiting

[info 2018/04/10 02:40:59.534 EDT event-server-1 <ReconnectThread> tid=0x217] 
GMSHealthMonitor serverSocketExecutor is terminated

[warning 2018/04/10 02:40:59.541 EDT event-server-1 <ReconnectThread> 
tid=0x217] Exception occurred while trying to connect the system during 
reconnect
org.apache.geode.security.AuthenticationRequiredException: Failed to find 
credentials from [host001(event-server-1:3525)<ec>:1026]
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
        at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
        at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
        at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
        at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
        at 
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)
        at 
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)
        at 
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
        at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
        at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
        at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
        at 
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
        at 
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
        at 
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
        at 
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1530)
        at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$0(GMSMembershipManager.java:2550)
        at java.lang.Thread.run(Thread.java:745)

Reply via email to