yes, this is continuously reproducible. Anyway will check the configs again and verify.
Regards, NuwanS. On Sat, May 16, 2015 at 2:34 PM, Dinusha Senanayaka <[email protected]> wrote: > HI Nuwan/ Asanthi, > > We configured 3-nodes cluster setup to test this scenario. Seems > everything works fine. "Member left" notification propagate to all other > nodes correctly. Also dep-sync works without issue with member left and > joining. > > Anyway, is this issue continuously re-producible in your setup ? Could see > some mismatch in the node that left and node that dep-sync has failed to > connect in your logs. > > Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 > Packet not sent to -> Address[192.168.48.5]:4000 > > May be some network issue in the machine caused this issue ? > > Btw, we don't need to add dep-sync for store/publisher. Only to GW cluster > is enough. > > Regards, > Dinusha. > > > On Thu, May 14, 2015 at 5:31 PM, Asanthi Kulasinghe <[email protected]> > wrote: > >> Looping in Azeez and Dinusha >> >> >> On Wed, May 13, 2015 at 2:33 PM, Nuwan Silva <[email protected]> wrote: >> >>> Hi All, >>> >>> I came across the following issue where we see a depsync error when a >>> single node in the cluster goes down and the other nodes not knowing it >>> till its restarted. >>> observation was that when a node (node 1) in the cluster goes down the >>> other nodes (node 2 and node 3) do not know that node (node 1) is down. So >>> it tries to synchronize its artifacts. >>> >>> further we observed that when the node which was down (node 1) restarts. >>> the other nodes find out that node1 was down. >>> >>> TID: [0] [AM] [2015-05-13 09:00:08,861] INFO >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>> TID: [0] [AM] [2015-05-13 09:00:08,877] INFO >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>> WKA member Host:192.168.48.2, Remote Host:null, Port: 4001, HTTP:8281, >>> HTTPS:8244, Domain: wso2.am.storepub.domain, Sub-domain:worker, Active:true >>> left cluster. >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>> TID: [0] [AM] [2015-05-13 09:00:08,890] INFO >>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - >>> Elected this member [10bb1dec-4d25-4966-a39a-eb42e42d1706] as the >>> Coordinator for the cluster [wso2.am.storepub.domain] >>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} >>> TID: [0] [AM] [2015-05-13 09:00:14,849] INFO >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} - >>> Member joined [bc438078-ee97-4643-aadb-cf7ba970dc7e]: /192.168.48.2:4001 >>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>> >>> >>> >>> *Depsync Error:* >>> >>> TID: [0] [AM] [2015-05-13 08:52:03,333] ERROR >>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - >>> Deployment synchronization commit for tenant -1234 failed >>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} >>> com.hazelcast.core.HazelcastException: >>> com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> >>> Address[192.168.48.5]:4000 >>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45) >>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:40) >>> at >>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:371) >>> at >>> com.hazelcast.map.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:215) >>> at >>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.containsKey(HazelcastDistributedMapProvider.java:119) >>> at >>> org.wso2.carbon.caching.impl.CacheImpl.containsKey(CacheImpl.java:237) >>> at >>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:291) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757) >>> at >>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262) >>> at >>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108) >>> at >>> org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96) >>> at >>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207) >>> at >>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet not >>> sent to -> Address[192.168.48.5]:4000 >>> at >>> com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:360) >>> at >>> com.hazelcast.spi.impl.BasicInvocation.access$900(BasicInvocation.java:62) >>> at >>> com.hazelcast.spi.impl.BasicInvocation$ReInvocationTask.run(BasicInvocation.java:549) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> at >>> com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) >>> at ------ End remote and begin local stack-trace ------.(Unknown >>> Source) >>> at >>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862) >>> at >>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795) >>> at >>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698) >>> at >>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676) >>> at >>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:369) >>> ... 22 more >>> >>> >>> Regards, >>> NuwanS. >>> -- >>> >>> >>> *Nuwan Silva* >>> *Senior Software Engineer - QA* >>> Mobile: +9477 980 4543 >>> >>> WSO2 Inc. >>> lean . enterprise . middlewear. >>> http://www.wso2.com >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Asanthi Kulasinghe* >> WSO2 Inc; http://www.wso2.com/. >> Mobile: +94777355522 >> >> >> > > > -- > Dinusha Dilrukshi > Associate Technical Lead > WSO2 Inc.: http://wso2.com/ > Mobile: +94725255071 > Blog: http://dinushasblog.blogspot.com/ > -- *Nuwan Silva* *Senior Software Engineer - QA* Mobile: +9477 980 4543 WSO2 Inc. lean . enterprise . middlewear. http://www.wso2.com
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
