Hi Avinash, The question to answer is “Why was a member removed from the cluster?” Some things to investigate:
- Insufficient heap for the data volume - Excessive GC causing the member to be unresponsive - OutOfMemory errors in the log - Overloaded CPU causing delayed heartbeats responses HTH, Anthony > On Jun 16, 2016, at 6:48 AM, Avinash Dongre <dongre.avin...@gmail.com> wrote: > > Hello All, > > I am getting following exception when I try to load my system with large > amount of data. > > My Setup Details: > 1 locator, 3 cacheservers with 8g and all the regions are disk persistence > enabled. ( All this is running on single AWS cluster node ) > > Please give me some clues what I am missing here ? > > > [severe 2016/06/16 12:51:15.552 UTC S1 <Notification Handler> tid=0x40] > Uncaught exception in thread Thread[Notification > Handler,10,ResourceListenerInvokerThreadGroup] > com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: > DistributedSystem is shutting down, caused by > com.gemstone.gemfire.ForcedDisconnectException: Member isn't responding to > heartbeat requests > at > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1719) > at > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1897) > at > com.gemstone.gemfire.distributed.internal.DistributionChannel.send(DistributionChannel.java:87) > at > com.gemstone.gemfire.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3427) > at > com.gemstone.gemfire.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3468) > at > com.gemstone.gemfire.distributed.internal.DistributionManager.putOutgoing(DistributionManager.java:1828) > at > com.gemstone.gemfire.internal.cache.control.ResourceAdvisor$ResourceProfileMessage.send(ResourceAdvisor.java:185) > at > com.gemstone.gemfire.internal.cache.control.ResourceAdvisor.updateRemoteProfile(ResourceAdvisor.java:448) > at > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.processLocalEvent(HeapMemoryMonitor.java:677) > at > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:485) > at > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor.updateStateAndSendEvent(HeapMemoryMonitor.java:448) > at > com.gemstone.gemfire.internal.cache.control.HeapMemoryMonitor$2.run(HeapMemoryMonitor.java:718) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.gemstone.gemfire.ForcedDisconnectException: Member isn't > responding to heartbeat requests > at > com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2551) > at > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:885) > at > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:578) > at > com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1540) > at > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1061) > at org.jgroups.JChannel.invokeCallback(JChannel.java:816) > at org.jgroups.JChannel.up(JChannel.java:741) > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) > at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) > at org.jgroups.protocols.FlowControl.up(FlowControl.java:392) > at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064) > at > org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779) > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426) > at > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:69) > at > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:74) > at org.jgroups.protocols.TP.passMessageUp(TP.java:1567) > at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1783) > at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) > at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1695) > at org.jgroups.protocols.TP.receive(TP.java:1620) > at > com.gemstone.gemfire.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:158) > at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) > ... 1 more
signature.asc
Description: Message signed with OpenPGP using GPGMail