Please upgrade to Hazelcast 3.4.2 On Mon, May 18, 2015 at 9:12 AM, Nuwan Silva <[email protected]> wrote:
> yes, this is continuously reproducible. Anyway will check the configs > again and verify. > > Regards, > NuwanS. > > On Sat, May 16, 2015 at 2:34 PM, Dinusha Senanayaka <[email protected]> > wrote: > >> HI Nuwan/ Asanthi, >> >> We configured 3-nodes cluster setup to test this scenario. Seems >> everything works fine. "Member left" notification propagate to all other >> nodes correctly. Also dep-sync works without issue with member left and >> joining. >> >> Anyway, is this issue continuously re-producible in your setup ? Could >> see some mismatch in the node that left and node that dep-sync has failed >> to connect in your logs. >> >> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 >> Packet not sent to -> Address[192.168.48.5]:4000 >> >> May be some network issue in the machine caused this issue ? >> >> Btw, we don't need to add dep-sync for store/publisher. Only to GW >> cluster is enough. >> >> Regards, >> Dinusha. >> >> >> On Thu, May 14, 2015 at 5:31 PM, Asanthi Kulasinghe <[email protected]> >> wrote: >> >>> Looping in Azeez and Dinusha >>> >>> >>> On Wed, May 13, 2015 at 2:33 PM, Nuwan Silva <[email protected]> wrote: >>> >>>> Hi All, >>>> >>>> I came across the following issue where we see a depsync error when a >>>> single node in the cluster goes down and the other nodes not knowing it >>>> till its restarted. >>>> observation was that when a node (node 1) in the cluster goes down the >>>> other nodes (node 2 and node 3) do not know that node (node 1) is down. So >>>> it tries to synchronize its artifacts. >>>> >>>> further we observed that when the node which was down (node 1) >>>> restarts. the other nodes find out that node1 was down. >>>> >>>> TID: [0] [AM] [2015-05-13 09:00:08,861] INFO >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>>> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>> TID: [0] [AM] [2015-05-13 09:00:08,877] INFO >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>>> WKA member Host:192.168.48.2, Remote Host:null, Port: 4001, HTTP:8281, >>>> HTTPS:8244, Domain: wso2.am.storepub.domain, Sub-domain:worker, Active:true >>>> left cluster. >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>> TID: [0] [AM] [2015-05-13 09:00:08,890] INFO >>>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - >>>> Elected this member [10bb1dec-4d25-4966-a39a-eb42e42d1706] as the >>>> Coordinator for the cluster [wso2.am.storepub.domain] >>>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} >>>> TID: [0] [AM] [2015-05-13 09:00:14,849] INFO >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>>> Member joined [bc438078-ee97-4643-aadb-cf7ba970dc7e]: / >>>> 192.168.48.2:4001 >>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>> >>>> >>>> >>>> *Depsync Error:* >>>> >>>> TID: [0] [AM] [2015-05-13 08:52:03,333] ERROR >>>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - >>>> Deployment synchronization commit for tenant -1234 failed >>>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} >>>> com.hazelcast.core.HazelcastException: >>>> com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> >>>> Address[192.168.48.5]:4000 >>>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45) >>>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:40) >>>> at >>>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:371) >>>> at >>>> com.hazelcast.map.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:215) >>>> at >>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.containsKey(HazelcastDistributedMapProvider.java:119) >>>> at >>>> org.wso2.carbon.caching.impl.CacheImpl.containsKey(CacheImpl.java:237) >>>> at >>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:291) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757) >>>> at >>>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262) >>>> at >>>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108) >>>> at >>>> org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96) >>>> at >>>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207) >>>> at >>>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet not >>>> sent to -> Address[192.168.48.5]:4000 >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:360) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation.access$900(BasicInvocation.java:62) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation$ReInvocationTask.run(BasicInvocation.java:549) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> at >>>> com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) >>>> at ------ End remote and begin local stack-trace ------.(Unknown >>>> Source) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698) >>>> at >>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676) >>>> at >>>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:369) >>>> ... 22 more >>>> >>>> >>>> Regards, >>>> NuwanS. >>>> -- >>>> >>>> >>>> *Nuwan Silva* >>>> *Senior Software Engineer - QA* >>>> Mobile: +9477 980 4543 >>>> >>>> WSO2 Inc. >>>> lean . enterprise . middlewear. >>>> http://www.wso2.com >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Asanthi Kulasinghe* >>> WSO2 Inc; http://www.wso2.com/. >>> Mobile: +94777355522 >>> >>> >>> >> >> >> -- >> Dinusha Dilrukshi >> Associate Technical Lead >> WSO2 Inc.: http://wso2.com/ >> Mobile: +94725255071 >> Blog: http://dinushasblog.blogspot.com/ >> > > > > -- > > > *Nuwan Silva* > *Senior Software Engineer - QA* > Mobile: +9477 980 4543 > > WSO2 Inc. > lean . enterprise . middlewear. > http://www.wso2.com > -- *Afkham Azeez* Director of Architecture; WSO2, Inc.; http://wso2.com Member; Apache Software Foundation; http://www.apache.org/ * <http://www.apache.org/>* *email: **[email protected]* <[email protected]> * cell: +94 77 3320919blog: **http://blog.afkham.org* <http://blog.afkham.org> *twitter: **http://twitter.com/afkham_azeez* <http://twitter.com/afkham_azeez> *linked-in: **http://lk.linkedin.com/in/afkhamazeez <http://lk.linkedin.com/in/afkhamazeez>* *Lean . Enterprise . Middleware*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
