Hello Team,
I faced a strange issue today where suspected member was forcefully
disconnected from distributed system due to lack of heartbeats.
But later on after so many attempts,
Ø Suspected member completely disconnected [DistributionManager closed]
Ø Attempted to reconnect to distributed system by starting membership
services, Jgroup channel, Gemfire P2P listener
Ø Failed due to following exception
org.apache.geode.security.AuthenticationRequiredException: Failed to find
credentials from [host001(event-server-1:3525)<ec>:1026]]
This looks very strange or may be issue. It was part of distributed system
using username/password and security manager from beginning. But after force
disconnection, when it attempted to join distributed system back, it was unable
to find credentials itself.
Could someone help me to validate it?
More,
Even though server was no more part of distributed system after all these
events, spring boot geode app was still running.
Shouldn't that also be stopped? I had to manually kill that for rectifying this.
I have attached detailed logs as well.
Topology
Version: Geode 1.1.1
Java: JDK 8
Platform: Red Hat Enterprise Linux
Host1:
Locator1
EventServer1 [Group = Events]
HopServer1 [Group = Hops]
Host2:
Locatro2
EventServer2 [Group = Events]
HopServer2 [Group = Hops]
Thanks,
Dharam
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
privilege, viruses and monitoring of electronic messages. If you are not the
intended recipient, please delete this message and notify the sender
immediately. Any unauthorized use is strictly prohibited.
[info 2018/04/10 02:39:19.324 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] received suspect message from
host002(event-server-2:1487)<ec><v31>:1026 for host001(Locator1:1134
:locator)<ec><v16>:1024: Member isn't responding to heartbeat requests
[warning 2018/04/10 02:39:20.688 EDT event-server-1 <Management Task> tid=0x58]
Attempting TCP/IP reconnect to host002(Locator2:3605:locator)<ec><v29>:1024
[warning 2018/04/10 02:39:20.707 EDT event-server-1 <P2P message reader for
host002(event-server-2:1487)<ec><v31>:1026 shared ordered uid=8 port=43352>
tid=0x189] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[warning 2018/04/10 02:39:33.778 EDT event-server-1 <Cache Server Load Polling
Thread> tid=0xf5] Attempting TCP/IP reconnect to
host002(Locator2:3605:locator)<ec><v29>:1024
[warning 2018/04/10 02:39:33.980 EDT event-server-1 <Queue Removal Thread>
tid=0x102] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[warning 2018/04/10 02:39:35.790 EDT event-server-1 <ServerConnection on port
40404 Thread 10> tid=0x124] 15 seconds have elapsed while waiting for replies:
<DistributedCacheOperation$CacheOperationReplyProcessor 513164 waiting for 1
replies from [host002(event-server-2:1487)<ec><v31>:1026]> on
host001(event-server-1:3525)<ec><v18>:1026 whose current membership list is:
[[host001(event-server-1:3525)<ec><v18>:1026,
host001(Locator1:1134:locator)<ec><v16>:1024,
host002(event-server-2:1487)<ec><v31>:1026,
host002(hp-server-2:1418)<ec><v30>:1025,
host002(Locator2:3605:locator)<ec><v29>:1024]]
[info 2018/04/10 02:39:36.837 EDT event-server-1 <P2P message reader for
host002(Locator2:3605:locator)<ec><v29>:1024 shared unordered uid=1 port=42338>
tid=0x17c] Performing final check for suspect member
host002(Locator2:3605:locator)<ec><v29>:1024 reason=member unexpectedly shut
down shared, unordered connection
[info 2018/04/10 02:39:36.851 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] Membership received a request to remove
host001(Locator1:1134:locator)<ec><v16>:1024 from
host002(Locator2:3605:locator)<ec><v29>:1024 reason=Member isn't responding to
heartbeat requests
[info 2018/04/10 02:39:36.852 EDT event-server-1 <ServerConnection on port
40404 Thread 2> tid=0x103] Server connection from
[identity(host002(1634:loner):34942:6531c5a4,connection=1; port=44006]:
connection disconnect detected by EOF.
[info 2018/04/10 02:39:37.365 EDT event-server-1 <P2P message reader for
host002(hp-server-2:1418)<ec><v30>:1025 shared unordered uid=1 port=43112>
tid=0x181] Performing final check for suspect member
host002(hp-server-2:1418)<ec><v30>:1025 reason=member unexpectedly shut down
shared, unordered connection
[info 2018/04/10 02:39:37.301 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] This member is becoming the membership
coordinator with address host001(event-server-1:3525)<ec><v18>:1026
[info 2018/04/10 02:39:37.281 EDT event-server-1 <Cache Server Load Polling
Thread> tid=0xf5] Successfully reconnected to member
host002(Locator2:3605:locator)<ec><v29>:1024
[info 2018/04/10 02:39:37.276 EDT event-server-1 <Management Task> tid=0x58]
Successfully reconnected to member host002(Locator2:3605:locator)<ec><v29>:1024
[info 2018/04/10 02:39:37.772 EDT event-server-1 <P2P message reader for
host002(event-server-2:1487)<ec><v31>:1026 shared ordered uid=8 port=43352>
tid=0x189] Successfully reconnected to member
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:37.820 EDT event-server-1 <P2P message reader for
host002(event-server-2:1487)<ec><v31>:1026 shared unordered uid=1 port=43268>
tid=0x186] Performing final check for suspect member
host002(event-server-2:1487)<ec><v31>:1026 reason=member unexpectedly shut down
shared, unordered connection
[info 2018/04/10 02:39:37.821 EDT event-server-1 <Geode Failure Detection
thread 19> tid=0x207] Performing final check for suspect member
host001(Locator1:1134:locator)<ec><v16>:1024 reason=Member isn't responding to
heartbeat requests
[warning 2018/04/10 02:39:37.930 EDT event-server-1 <ServerConnection on port
40404 Thread 2> tid=0x103] ClientHealthMonitor: Unregistering client with
member id identity(host002(1634:loner):34942:6531c5a4,connection=1 due to: The
connection has been reset while reading the header
[info 2018/04/10 02:39:37.930 EDT event-server-1 <Queue Removal Thread>
tid=0x102] Successfully reconnected to member
host002(event-server-2:1487)<ec><v31>:1026
[warning 2018/04/10 02:39:38.039 EDT event-server-1 <Queue Removal Thread>
tid=0x102] Attempting TCP/IP reconnect to
host002(hp-server-2:1418)<ec><v30>:1025
[info 2018/04/10 02:39:38.725 EDT event-server-1 <Queue Removal Thread>
tid=0x102] Successfully reconnected to member
host002(hp-server-2:1418)<ec><v30>:1025
[info 2018/04/10 02:39:38.769 EDT event-server-1 <Geode Failure Detection
thread 16> tid=0x1fa] received suspect message from
host001(event-server-1:3525)<ec><v18>:1026 for
host002(event-server-2:1487)<ec><v31>:1026: Member isn't responding to
heartbeat requests
[info 2018/04/10 02:39:38.830 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] Server connection from
[identity(host001(8115:loner):50832:190583a4,connection=1; port=47752]:
connection disconnect detected by EOF.
[warning 2018/04/10 02:39:38.831 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] ClientHealthMonitor: Unregistering client with
member id identity(host001(8115:loner):50832:190583a4,connection=1 due to: The
connection has been reset while reading the header
[warning 2018/04/10 02:39:39.215 EDT event-server-1 <Management Task> tid=0x58]
Attempting TCP/IP reconnect to host002(Locator2:3605:locator)<ec><v29>:1024
[info 2018/04/10 02:39:39.265 EDT event-server-1 <P2P message reader for
host002(event-server-2:1487)<ec><v31>:1026 shared unordered uid=1 port=43268>
tid=0x186] Final check passed for suspect member
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:39.269 EDT event-server-1 <P2P message reader for
host002(Locator2:3605:locator)<ec><v29>:1024 shared unordered uid=1 port=42338>
tid=0x17c] Final check passed for suspect member
host002(Locator2:3605:locator)<ec><v29>:1024
[info 2018/04/10 02:39:39.265 EDT event-server-1 <P2P message reader for
host002(hp-server-2:1418)<ec><v30>:1025 shared unordered uid=1 port=43112>
tid=0x181] Final check passed for suspect member
host002(hp-server-2:1418)<ec><v30>:1025
[warning 2018/04/10 02:39:39.469 EDT event-server-1 <Cache Server Load Polling
Thread> tid=0xf5] Attempting TCP/IP reconnect to
host002(Locator2:3605:locator)<ec><v29>:1024
[warning 2018/04/10 02:39:39.484 EDT event-server-1 <Queue Removal Thread>
tid=0x102] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:39.484 EDT event-server-1 <Geode Failure Detection
thread 19> tid=0x207] Final check passed for suspect member
host001(Locator1:1134:locator)<ec><v16>:1024
[warning 2018/04/10 02:39:43.118 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[severe 2018/04/10 02:39:43.301 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] This member is no longer in the membership
view. My ID is host001(event-server-1:3525)<ec><v18>:1026 and the new view is
View[host002(Locator2:3605:locator)<ec><v29>:1024|38] members:
[host002(Locator2:3605:locator)<ec><v29>:1024,
host002(hp-server-2:1418)<ec><v30>:1025{lead},
host002(event-server-2:1487)<ec><v31>:1026] crashed:
[host001(Locator1:1134:locator)<ec><v16>:1024,
host001(event-server-1:3525)<ec><v18>:1026]
[info 2018/04/10 02:39:43.325 EDT event-server-1 <Geode Membership View
Creator> tid=0x20d] View Creator thread is starting
[warning 2018/04/10 02:39:43.434 EDT event-server-1 <Management Task> tid=0x58]
Attempting TCP/IP reconnect to host002(Locator2:3605:locator)<ec><v29>:1024
[info 2018/04/10 02:39:43.544 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] Successfully reconnected to member
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:43.614 EDT event-server-1 <Geode Membership View
Creator> tid=0x20d] preparing new view
View[host001(event-server-1:3525)<ec><v18>:1026|46] members:
[host001(event-server-1:3525)<ec><v18>:1026{lead},
host002(Locator2:3605:locator)<ec><v29>:1024,
host002(hp-server-2:1418)<ec><v30>:1025,
host002(event-server-2:1487)<ec><v31>:1026] crashed:
[host001(Locator1:1134:locator)<ec><v16>:1024]
failure detection ports: 12988 44464 27210 21079
[warning 2018/04/10 02:39:43.658 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:43.841 EDT event-server-1 <ServerConnection on port
40404 Thread 6> tid=0x10c] Successfully reconnected to member
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:43.993 EDT event-server-1 <Management Task> tid=0x58]
Successfully reconnected to member host002(Locator2:3605:locator)<ec><v29>:1024
[warning 2018/04/10 02:39:43.992 EDT event-server-1 <ServerConnection on port
40404 Thread 7> tid=0x10d] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:44.016 EDT event-server-1 <ServerConnection on port
40404 Thread 7> tid=0x10d] Ending reconnect attempt to
host002(event-server-2:1487)<ec><v31>:1026 because shutdown has started.
[warning 2018/04/10 02:39:44.089 EDT event-server-1 <ServerConnection on port
40404 Thread 2> tid=0x103] Attempting TCP/IP reconnect to
host002(event-server-2:1487)<ec><v31>:1026
[info 2018/04/10 02:39:44.089 EDT event-server-1 <ServerConnection on port
40404 Thread 2> tid=0x103] Ending reconnect attempt to
host002(event-server-2:1487)<ec><v31>:1026 because shutdown has started.
[info 2018/04/10 02:39:44.199 EDT event-server-1 <Queue Removal Thread>
tid=0x102] The QueueRemovalThread is done.
[severe 2018/04/10 02:39:44.379 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] Membership service failure: This node is no
longer in the membership view
org.apache.geode.ForcedDisconnectException: This node is no longer in the
membership view
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2520)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:998)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processViewMessage(GMSJoinLeave.java:984)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1690)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1286)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
at
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
at org.jgroups.protocols.TP.receive(TP.java:1654)
at
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
at java.lang.Thread.run(Thread.java:745)
[info 2018/04/10 02:39:44.683 EDT event-server-1 <unicast
receiver,host001-11901> tid=0x33] CacheServer configuration saved
[info 2018/04/10 02:39:44.848 EDT event-server-1 <Geode Membership View
Creator> tid=0x20d] finished waiting for responses to view preparation
[info 2018/04/10 02:39:45.815 EDT event-server-1 <DisconnectThread> tid=0x217]
Stopping membership services
[info 2018/04/10 02:39:45.966 EDT event-server-1 <DisconnectThread> tid=0x217]
GMSHealthMonitor server socket is closed in stopServices().
[warning 2018/04/10 02:39:46.148 EDT event-server-1 <ServerConnection on port
40404 Thread 2> tid=0x103] Server connection from
[identity(host001(8115:loner):50832:190583a4,connection=2; port=49136]:
Unexpected Exception
org.apache.geode.distributed.DistributedSystemDisconnectedException:
DistributedSystem is shutting down, caused by
org.apache.geode.ForcedDisconnectException: This node is no longer in the
membership view
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1700)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
at
org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
at
org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
at
org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
at
org.apache.geode.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1832)
at
org.apache.geode.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:505)
at
org.apache.geode.internal.cache.DistributedRegion.distributeDestroy(DistributedRegion.java:1701)
at
org.apache.geode.internal.cache.DistributedRegion.basicDestroyPart3(DistributedRegion.java:1692)
at
org.apache.geode.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1519)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6778)
at
org.apache.geode.internal.cache.LocalRegion.mapDestroy(LocalRegion.java:6755)
at
org.apache.geode.internal.cache.LocalRegionDataView.destroyExistingEntry(LocalRegionDataView.java:55)
at
org.apache.geode.internal.cache.LocalRegion.basicDestroy(LocalRegion.java:6717)
at
org.apache.geode.internal.cache.DistributedRegion.basicDestroy(DistributedRegion.java:1662)
at
org.apache.geode.internal.cache.LocalRegion.basicBridgeDestroy(LocalRegion.java:5614)
at
org.apache.geode.internal.cache.tier.sockets.command.Destroy65.cmdExecute(Destroy65.java:239)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1138)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.geode.ForcedDisconnectException: This node is no longer
in the membership view
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2520)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:998)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processViewMessage(GMSJoinLeave.java:984)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1690)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1286)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1070)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:785)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:74)
at
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1601)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1817)
at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1729)
at org.jgroups.protocols.TP.receive(TP.java:1654)
at
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:160)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
... 1 more
[info 2018/04/10 02:39:46.400 EDT event-server-1 <Geode Failure Detection
Server thread 0> tid=0x37] GMSHealthMonitor server thread exiting
[info 2018/04/10 02:39:46.419 EDT event-server-1 <DisconnectThread> tid=0x217]
GMSHealthMonitor serverSocketExecutor is terminated
[info 2018/04/10 02:39:48.577 EDT event-server-1 <Cache Server Selector
/0.0.0.0:40404 local port: 40404> tid=0xfa] Cache server on port 40,404 is
shutting down.
[info 2018/04/10 02:39:49.200 EDT event-server-1 <ReconnectThread> tid=0x217]
Disconnecting old DistributedSystem to prepare for a reconnect attempt
[info 2018/04/10 02:39:49.757 EDT event-server-1 <ReconnectThread> tid=0x217]
GemFireCache[id = 1710483461; isClosing = true; isShutDownAll = false; created
= Sun Apr 08 05:04:20 EDT 2018; server = false; copyOnRead = false; lockLease =
120; lockTimeout = 60]: Now closing.
[info 2018/04/10 02:39:56.831 EDT event-server-1 <ReconnectThread> tid=0x217]
Created oplog#2 krf for disk store events_disk_store.
[info 2018/04/10 02:39:56.847 EDT event-server-1 <ReconnectThread> tid=0x217]
Created oplog#2 krf for disk store pdx_metadata_diskstore.
[info 2018/04/10 02:39:58.157 EDT event-server-1 <ReconnectThread> tid=0x217]
Rest Server on port 9,301 is shutting down
[info 2018/04/10 02:39:58.176 EDT event-server-1 <ReconnectThread> tid=0x217]
Stopping the HTTP service...
[info 2018/04/10 02:39:59.229 EDT event-server-1 <ReconnectThread> tid=0x217]
Shutting down DistributionManager host001(event-server-1:3525)<ec><v18>:1026.
At least one Exception occurred.
[info 2018/04/10 02:39:59.420 EDT event-server-1 <ReconnectThread> tid=0x217]
Now closing distribution for host001(event-server-1:3525)<ec><v18>:1026
[info 2018/04/10 02:39:59.423 EDT event-server-1 <ReconnectThread> tid=0x217]
DistributionManager stopped in 193ms.
[info 2018/04/10 02:39:59.423 EDT event-server-1 <ReconnectThread> tid=0x217]
Marking DistributionManager host001(event-server-1:3525)<ec><v18>:1026 as
closed.
[info 2018/04/10 02:40:59.470 EDT event-server-1 <ReconnectThread> tid=0x217]
Startup Configuration:
### GemFire Properties defined with api ###
ack-severe-alert-threshold=0
ack-wait-threshold=15
archive-disk-space-limit=0
archive-file-size-limit=0
async-distribution-timeout=0
async-max-queue-size=8
async-queue-timeout=60000
bind-address=
cache-xml-file=cache.xml
cluster-configuration-dir=
cluster-ssl-ciphers=any
cluster-ssl-enabled=false
cluster-ssl-keystore=
cluster-ssl-keystore-password=********
cluster-ssl-keystore-type=
cluster-ssl-protocols=any
cluster-ssl-require-authentication=true
cluster-ssl-truststore=
cluster-ssl-truststore-password=********
conflate-events=server
conserve-sockets=true
delta-propagation=true
deploy-working-dir=/home/geode/deploy
disable-auto-reconnect=false
disable-tcp=false
distributed-system-id=-1
distributed-transactions=false
durable-client-id=
durable-client-timeout=300
enable-cluster-configuration=true
enable-network-partition-detection=false
enable-time-statistics=false
enforce-unique-host=false
gateway-ssl-ciphers=any
gateway-ssl-enabled=false
gateway-ssl-keystore=
gateway-ssl-keystore-password=********
gateway-ssl-keystore-type=
gateway-ssl-protocols=any
gateway-ssl-require-authentication=true
gateway-ssl-truststore=
gateway-ssl-truststore-password=********
groups=INSTRUMENTS
http-service-bind-address=host001
http-service-port=9301
http-service-ssl-ciphers=any
http-service-ssl-enabled=false
http-service-ssl-keystore=
http-service-ssl-keystore-password=********
http-service-ssl-keystore-type=
http-service-ssl-protocols=any
http-service-ssl-require-authentication=false
http-service-ssl-truststore=
http-service-ssl-truststore-password=********
jmx-manager=false
jmx-manager-access-file=
jmx-manager-bind-address=
jmx-manager-hostname-for-clients=
jmx-manager-http-port=9301
jmx-manager-password-file=********
jmx-manager-port=1099
jmx-manager-ssl-ciphers=any
jmx-manager-ssl-enabled=false
jmx-manager-ssl-keystore=
jmx-manager-ssl-keystore-password=********
jmx-manager-ssl-keystore-type=
jmx-manager-ssl-protocols=any
jmx-manager-ssl-require-authentication=true
jmx-manager-ssl-truststore=
jmx-manager-ssl-truststore-password=********
jmx-manager-start=false
jmx-manager-update-rate=2000
load-cluster-configuration-from-dir=false
locator-wait-time=0
locators=host001.company.com[10334],host002.company.com[10334]
lock-memory=false
log-disk-space-limit=2048
log-file=/home/logs/current/event-server-1/event-server-1.log
log-file-size-limit=100
log-level=info
max-num-reconnect-tries=3
max-wait-time-reconnect=60000
mcast-address=239.192.81.1
mcast-flow-control=1048576, 0.25, 5000
mcast-port=0
mcast-recv-buffer-size=1048576
mcast-send-buffer-size=65535
mcast-ttl=32
member-timeout=5000
membership-port-range=1024-65535
memcached-bind-address=
memcached-port=0
memcached-protocol=ASCII
name=event-server-1
off-heap-memory-size=
redis-bind-address=
redis-password=********
redis-port=0
redundancy-zone=
remote-locators=
remove-unresponsive-client=false
roles=
security-client-accessor=********
security-client-accessor-pp=********
security-client-auth-init=********
security-client-authenticator=********
security-client-dhalgo=********
security-log-file=********
security-log-level=********
security-manager=********
security-peer-auth-init=********
security-peer-authenticator=********
security-peer-verifymember-timeout=********
security-post-processor=********
security-udp-dhalgo=********
server-bind-address=
server-ssl-ciphers=any
server-ssl-enabled=false
server-ssl-keystore=
server-ssl-keystore-password=********
server-ssl-keystore-type=
server-ssl-protocols=any
server-ssl-require-authentication=true
server-ssl-truststore=
server-ssl-truststore-password=********
socket-buffer-size=32768
socket-lease-time=60000
ssl-ciphers=any
ssl-cluster-alias=
ssl-default-alias=
ssl-enabled-components=
ssl-gateway-alias=
ssl-jmx-alias=
ssl-keystore=
ssl-keystore-password=********
ssl-keystore-type=
ssl-locator-alias=
ssl-protocols=any
ssl-require-authentication=true
ssl-server-alias=
ssl-truststore=
ssl-truststore-password=********
ssl-web-alias=
ssl-web-require-authentication=false
start-dev-rest-api=true
start-locator=
statistic-archive-file=
statistic-sample-rate=1000
statistic-sampling-enabled=true
tcp-port=0
udp-fragment-size=60000
udp-recv-buffer-size=1048576
udp-send-buffer-size=65535
use-cluster-configuration=true
user-command-packages=
[info 2018/04/10 02:40:59.426 EDT event-server-1 <ReconnectThread> tid=0x217]
Attempting to reconnect to the distributed system. This is attempt #1.
[info 2018/04/10 02:40:59.489 EDT event-server-1 <ReconnectThread> tid=0x217]
Starting membership services
[info 2018/04/10 02:40:59.497 EDT event-server-1 <ReconnectThread> tid=0x217]
JGroups channel reinitialized (took 8ms)
[info 2018/04/10 02:40:59.499 EDT event-server-1 <ReconnectThread> tid=0x217]
GemFire P2P Listener started on host001.company.com/169.87.179.47:34532
[info 2018/04/10 02:40:59.505 EDT event-server-1 <Geode Failure Detection
Server thread 0> tid=0x232] Started failure detection server thread on
/169.87.179.47:62294.
[info 2018/04/10 02:40:59.516 EDT event-server-1 <ReconnectThread> tid=0x217]
Attempting to join the distributed system through coordinator
host002(Locator2:3605:locator)<ec><v29>:1024 using address
host001(event-server-1:3525)<ec>:1026
[info 2018/04/10 02:40:59.522 EDT event-server-1 <ReconnectThread> tid=0x217]
Stopping membership services
[info 2018/04/10 02:40:59.534 EDT event-server-1 <ReconnectThread> tid=0x217]
GMSHealthMonitor server socket is closed in stopServices().
[info 2018/04/10 02:40:59.534 EDT event-server-1 <Geode Failure Detection
Server thread 0> tid=0x232] GMSHealthMonitor server thread exiting
[info 2018/04/10 02:40:59.534 EDT event-server-1 <ReconnectThread> tid=0x217]
GMSHealthMonitor serverSocketExecutor is terminated
[warning 2018/04/10 02:40:59.541 EDT event-server-1 <ReconnectThread>
tid=0x217] Exception occurred while trying to connect the system during
reconnect
org.apache.geode.security.AuthenticationRequiredException: Failed to find
credentials from [host001(event-server-1:3525)<ec>:1026]
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
at
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)
at
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)
at
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1530)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$0(GMSMembershipManager.java:2550)
at java.lang.Thread.run(Thread.java:745)
