Hi Udo, Sure! I have attached demo project with apache-geode 1.8.0 & spring-data-geode 2.1.4.RELEASE. Hope that helps!
Download demo.tar from -> https://drive.google.com/open?id=10j3NQEkb74n9tNBuWxza9J772p7IYg-7 *Steps:* 1. Start locator $GFSH start locator --name=locator1 --port=10334 --J=-Denable-network-partition-detection=false --log-level=config 2. Start server1 java -jar demo-0.0.1-SNAPSHOT.jar -Ddemo.name=demo-server-1 -Ddemo.port=40440 3. Start server2 java -jar demo-0.0.1-SNAPSHOT.jar -Ddemo.name=demo-server-2 -Ddemo.port=40441 4. Execute below command to suspend server 'demo-server-2' PID=`ps auxwww | fgrep 'java' | fgrep 'demo-server-2' | awk '{print $2}'` ; kill -STOP $PID 5. Make sure server 'demo-server-2' is thrown out of distributed system from logs/pulse 6. Resume server 'demo-server-2' and let it bootstrap and rejoin distributed system (This will take few seconds so be patient!) PID=`ps auxwww | fgrep 'java' | fgrep 'demo-server-2' | awk '{print $2}'` ; kill -CONT $PID 7. Let the member make at least 2 reconnect attempt (You will see all possible track traces I provided earlier) Thanks, - Dharam Thacker On Fri, Feb 1, 2019 at 11:23 PM Udo Kohlmeyer <u...@apache.org> wrote: > Hi there Dharam, > > Given that you are testing with Geode 1.6 + 1.8 and seemingly you are > using Spring Data Geode. Could you possibly provide a little more > information on HOW you are starting the servers? Also what version Spring > Data Geode and maybe, if you could, a simplified configuration / java > classes that cause this to happen. > > --Udo > On 2/1/19 04:37, Thacker, Dharam wrote: > > John/Bruce, > > > > It’s the same behavior even with “MappingPdxSerializer” for “Force > Reconnect” L > > > > I would really appreciate if we can address into GEODE-1.9.0 release J > > > > It tries for 6 attempts to reconnect distributed system. In first attempt > it still complains for “pdxSerializer” issue and in later attempts it > always gives, > > > > [info 2019-02-01 18:01:33.878 IST <ReconnectThread> > internal.ClusterDistributionManager tid=142] Serial Queue info : > THROTTLE_PERCENT: 0.75 SERIAL_QUEUE_BYTE_LIMIT :41943040 > SERIAL_QUEUE_THROTTLE :31457280 TOTAL_SERIAL_QUEUE_BYTE_LIMIT :83886080 > TOTAL_SERIAL_QUEUE_THROTTLE :31457280 SERIAL_QUEUE_SIZE_LIMIT :20000 > SERIAL_QUEUE_SIZE_THROTTLE :15000 > > > > [info 2019-02-01 18:01:33.883 IST <ReconnectThread> gms.Services tid=142] > Starting membership services > > > > [fatal 2019-02-01 18:01:33.883 IST <ReconnectThread> gms.Services tid=142] > Unexpected exception while booting membership services > > *java.lang.NullPointerException* > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress( > *JGroupsMessenger.java:483*) > > … > > > > > > @EnablePdx(readSerialized = *true*, serializerBeanName = > "mappingPdxSerializer") > > > > @Bean > > public MappingPdxSerializer mappingPdxSerializer() { > > MappingPdxSerializer mappingPdxSerializer = new > MappingPdxSerializer(); > > Map<Class<?>, EntityInstantiator> customInstantiators = > new HashMap<>(); > > EntityInstantiator instantiator = > ReflectionEntityInstantiator.INSTANCE; > > customInstantiators.put(A1.class, instantiator); > > customInstantiators.put(A1Audit.class, instantiator); > > customInstantiators.put(B1.class, instantiator); > > customInstantiators.put(B1Audit.class, instantiator); > > return mappingPdxSerializer; > > } > > > > > > 1 more issue with MappingPdxSerializer, > > Class A1 à region A1 > > > > a1Reposiroty.findAll() à This should give me List<A1> as a standard > contact. > > > > Ø Works with ReflectionBasedAutoSerializer > > Ø Fails with “MappingPdxSerializer” (It gives List<PdxInstanceImp> and > internally you have to call pdxInstance.getObject() to get real instance of > A1) > > > > > > > > Thanks, > > Dharam > > > > *From:* Thacker, Dharam > *Sent:* Friday, February 01, 2019 12:58 PM > *To:* 'user@geode.apache.org' <user@geode.apache.org> > *Subject:* RE: NullPointerException with GMS services (Geode 1.6.0) > > > > Hi John/Bruce, > > > > Typo in email was by me but code does not have that. > > > > Here is the detailed logs explaining what might have happened and looks > like a CRITICAL BUG only in both (1.6.0) and (1.8.0) as well. > > > > [Attached email thread for GEODE 1.8.0 as well] > > > > *ISSUES:* > > > > Ø 1 common scenario which fails – Member is unable to join distributed > system back after FORCE DICONNECT > > > > Ø Why does PDXSerializer play a role while boot strapping cache member? > (Note : No disk persistence is being used but only > ReflectionBasedAutoSerializer) > > > > Ø GMS is throwing NullPointer exception in this situation > > > > Ø Same error messages and stack trace for GMS is being printed once with > fatal/error/warn log levels which does not look consistent. > > > > > > [Pending] : Dharam to find out if that’s the same issue with > MappingPdxSerializer as suggested by John > > > > Current logs both for GEODE – 1.6.0 & GEODE 1.8.0 > > > > > > [warn 2019-02-01 00:05:29.000 EST <StatSampler> > statistics.HostStatSampler] Statistics sampling thread detected a wakeup > delay of 16617 ms, indicating a possible resource issue. Check the GC, > memory, and CPU statistics. > > > > [fatal 2019-02-01 00:05:29.937 EST <unicast receiver,iaasn00005748-49024> > gms.Services] Membership service failure: Failed to acknowledge a new > membership view and then failed tcp/ip connection attempt > > org.apache.geode.ForcedDisconnectException: Failed to acknowledge a new > membership view and then failed tcp/ip connection attempt > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2503) > > at > org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1049) > > at > org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:654) > > at > org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1810) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1301) > > at org.jgroups.JChannel.invokeCallback(JChannel.java:816) > > at org.jgroups.JChannel.up(JChannel.java:741) > > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030) > > at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) > > at org.jgroups.protocols.FlowControl.up(FlowControl.java:390) > > at > org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077) > > at > org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792) > > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72) > > at org.jgroups.protocols.TP.passMessageUp(TP.java:1658) > > at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876) > > at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10) > > at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789) > > at org.jgroups.protocols.TP.receive(TP.java:1714) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152) > > at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701) > > at java.lang.Thread.run(Thread.java:745) > > > > [error 2019-02-01 00:06:35.651 EST <ReconnectThread> > cache.GemFireCacheImpl] org.apache.geode.cache.CacheClosedException: Could > not PDX serialize because the cache was closed > > > > [warn 2019-02-01 00:06:35.652 EST <ReconnectThread> > internal.InternalDistributedSystem] Exception occurred while trying to > create the cache during reconnect > > org.apache.geode.cache.CacheClosedException: Could not PDX serialize > because the cache was closed > > at > org.apache.geode.pdx.internal.TypeRegistry.getPdxSerializer(TypeRegistry.java:317) > > at > org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1648) > > at > org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1548) > > at > org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2200) > > at > org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2952) > > at > org.apache.geode.internal.cache.MemberFunctionStreamingMessage.toData(MemberFunctionStreamingMessage.java:315) > > at > org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2398) > > at > org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1517) > > at > org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:234) > > at > org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:394) > > at > org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:251) > > at > org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:616) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1686) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1864) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2860) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2780) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2819) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523) > > at > org.apache.geode.internal.cache.execute.StreamingFunctionOperation.getFunctionResultFrom(StreamingFunctionOperation.java:107) > > at > org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:151) > > at > org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:189) > > at > org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:392) > > at > org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromOneLocator(ClusterConfigurationLoader.java:312) > > at > org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromLocators(ClusterConfigurationLoader.java:282) > > at > org.apache.geode.internal.cache.GemFireCacheImpl.requestSharedConfiguration(GemFireCacheImpl.java:1066) > > at > org.apache.geode.internal.cache.GemFireCacheImpl.<init>(GemFireCacheImpl.java:857) > > at > org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:794) > > at > org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:773) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2765) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531) > > at java.lang.Thread.run(Thread.java:745) > > > > [fatal 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services] > Unexpected exception while booting membership services > > java.lang.NullPointerException > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361) > > at > org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146) > > at > org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105) > > at > org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342) > > at > org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531) > > at java.lang.Thread.run(Thread.java:745) > > > > [error 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services] > Unexpected problem starting up membership services > > java.lang.NullPointerException > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361) > > at > org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146) > > at > org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105) > > at > org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342) > > at > org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Thanks, > > Dharam > > > > *From:* John Blum [mailto:jb...@pivotal.io <jb...@pivotal.io>] > *Sent:* Tuesday, January 29, 2019 3:49 AM > *To:* user@geode.apache.org > *Subject:* Re: NullPointerException with GMS services (Geode 1.6.0) > > > > Although, there is an apparent problem with your Locator configuration > property, which can/will lead to an NPE thrown by Geode in this case (as I > reported in https://issues.apache.org/jira/browse/GEODE-6153, which is > not exactly the same as your problem), as Bruce points out, this is > occurring during reconnect. > > > > If the default value of the Locator property had been used (which it > apparently wasn't in this case), then the node would have failed even on > initial startup. So, something else is going wrong here. > > > > On Mon, Jan 28, 2019 at 10:32 AM Bruce Schuchardt <bschucha...@pivotal.io> > wrote: > > These NPEs are probably not your main problem since their occurring in a > Reconnect Thread. Something bad happened when you tried to start these > caches. They joined the cluster and then were kicked out for some reason. > The log file for that process will likely tell you what the problem was. > Post it or PM it to me if you want help. > > On 1/27/19 10:56 PM, Thacker, Dharam wrote: > > java.lang.NullPointerException > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:468) > > at > org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:355) > > at > org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:157) > > at > org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:106) > > at > org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1027) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1061) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:554) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:763) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:355) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:341) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:335) > > at > org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:211) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2736) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2560) > > at > org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1041) > > at > org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:4033) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1554) > > at > org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$1(GMSMembershipManager.java:2561) > > at java.lang.Thread.run(Thread.java:745) > > > > > -- > > -John > > john.blum10101 (skype) > > This message is confidential and subject to terms at: https:// > www.jpmorgan.com/emaildisclaimer including on confidentiality, legal > privilege, viruses and monitoring of electronic messages. If you are not > the intended recipient, please delete this message and notify the sender > immediately. Any unauthorized use is strictly prohibited. > >
