HI Nuwan/ Asanthi, We configured 3-nodes cluster setup to test this scenario. Seems everything works fine. "Member left" notification propagate to all other nodes correctly. Also dep-sync works without issue with member left and joining.
Anyway, is this issue continuously re-producible in your setup ? Could see some mismatch in the node that left and node that dep-sync has failed to connect in your logs. Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 Packet not sent to -> Address[192.168.48.5]:4000 May be some network issue in the machine caused this issue ? Btw, we don't need to add dep-sync for store/publisher. Only to GW cluster is enough. Regards, Dinusha. On Thu, May 14, 2015 at 5:31 PM, Asanthi Kulasinghe <[email protected]> wrote: > Looping in Azeez and Dinusha > > > On Wed, May 13, 2015 at 2:33 PM, Nuwan Silva <[email protected]> wrote: > >> Hi All, >> >> I came across the following issue where we see a depsync error when a >> single node in the cluster goes down and the other nodes not knowing it >> till its restarted. >> observation was that when a node (node 1) in the cluster goes down the >> other nodes (node 2 and node 3) do not know that node (node 1) is down. So >> it tries to synchronize its artifacts. >> >> further we observed that when the node which was down (node 1) restarts. >> the other nodes find out that node1 was down. >> >> TID: [0] [AM] [2015-05-13 09:00:08,861] INFO >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >> TID: [0] [AM] [2015-05-13 09:00:08,877] INFO >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >> WKA member Host:192.168.48.2, Remote Host:null, Port: 4001, HTTP:8281, >> HTTPS:8244, Domain: wso2.am.storepub.domain, Sub-domain:worker, Active:true >> left cluster. >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >> TID: [0] [AM] [2015-05-13 09:00:08,890] INFO >> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - >> Elected this member [10bb1dec-4d25-4966-a39a-eb42e42d1706] as the >> Coordinator for the cluster [wso2.am.storepub.domain] >> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} >> TID: [0] [AM] [2015-05-13 09:00:14,849] INFO >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >> Member joined [bc438078-ee97-4643-aadb-cf7ba970dc7e]: /192.168.48.2:4001 >> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >> >> >> >> *Depsync Error:* >> >> TID: [0] [AM] [2015-05-13 08:52:03,333] ERROR >> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - >> Deployment synchronization commit for tenant -1234 failed >> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} >> com.hazelcast.core.HazelcastException: >> com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> >> Address[192.168.48.5]:4000 >> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45) >> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:40) >> at >> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:371) >> at >> com.hazelcast.map.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:215) >> at >> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.containsKey(HazelcastDistributedMapProvider.java:119) >> at >> org.wso2.carbon.caching.impl.CacheImpl.containsKey(CacheImpl.java:237) >> at >> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:291) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757) >> at >> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262) >> at >> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108) >> at >> org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96) >> at >> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207) >> at >> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet not >> sent to -> Address[192.168.48.5]:4000 >> at >> com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:360) >> at >> com.hazelcast.spi.impl.BasicInvocation.access$900(BasicInvocation.java:62) >> at >> com.hazelcast.spi.impl.BasicInvocation$ReInvocationTask.run(BasicInvocation.java:549) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> at >> com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) >> at ------ End remote and begin local stack-trace ------.(Unknown >> Source) >> at >> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862) >> at >> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795) >> at >> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698) >> at >> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676) >> at >> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:369) >> ... 22 more >> >> >> Regards, >> NuwanS. >> -- >> >> >> *Nuwan Silva* >> *Senior Software Engineer - QA* >> Mobile: +9477 980 4543 >> >> WSO2 Inc. >> lean . enterprise . middlewear. >> http://www.wso2.com >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Asanthi Kulasinghe* > WSO2 Inc; http://www.wso2.com/. > Mobile: +94777355522 > > > -- Dinusha Dilrukshi Associate Technical Lead WSO2 Inc.: http://wso2.com/ Mobile: +94725255071 Blog: http://dinushasblog.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
