Hi All,
I came across the following issue where we see a depsync error when a
single node in the cluster goes down and the other nodes not knowing it
till its restarted.
observation was that when a node (node 1) in the cluster goes down the
other nodes (node 2 and node 3) do not know that node (node 1) is down. So
it tries to synchronize its artifacts.
further we observed that when the node which was down (node 1) restarts.
the other nodes find out that node1 was down.
TID: [0] [AM] [2015-05-13 09:00:08,861] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [0] [AM] [2015-05-13 09:00:08,877] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
WKA member Host:192.168.48.2, Remote Host:null, Port: 4001, HTTP:8281,
HTTPS:8244, Domain: wso2.am.storepub.domain, Sub-domain:worker, Active:true
left cluster.
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
TID: [0] [AM] [2015-05-13 09:00:08,890] INFO
{org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} -
Elected this member [10bb1dec-4d25-4966-a39a-eb42e42d1706] as the
Coordinator for the cluster [wso2.am.storepub.domain]
{org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [0] [AM] [2015-05-13 09:00:14,849] INFO
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -
Member joined [bc438078-ee97-4643-aadb-cf7ba970dc7e]: /192.168.48.2:4001
{org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}
*Depsync Error:*
TID: [0] [AM] [2015-05-13 08:52:03,333] ERROR
{org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -
Deployment synchronization commit for tenant -1234 failed
{org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}
com.hazelcast.core.HazelcastException:
com.hazelcast.spi.exception.RetryableIOException: Packet not sent to ->
Address[192.168.48.5]:4000
at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45)
at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:40)
at
com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:371)
at
com.hazelcast.map.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:215)
at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.containsKey(HazelcastDistributedMapProvider.java:119)
at
org.wso2.carbon.caching.impl.CacheImpl.containsKey(CacheImpl.java:237)
at
org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:291)
at
org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777)
at
org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60)
at
org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760)
at
org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at
org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757)
at
org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262)
at
org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108)
at
org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96)
at
org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207)
at
org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet not
sent to -> Address[192.168.48.5]:4000
at
com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:360)
at
com.hazelcast.spi.impl.BasicInvocation.access$900(BasicInvocation.java:62)
at
com.hazelcast.spi.impl.BasicInvocation$ReInvocationTask.run(BasicInvocation.java:549)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at
com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
at ------ End remote and begin local stack-trace ------.(Unknown Source)
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862)
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795)
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698)
at
com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at
com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:369)
... 22 more
Regards,
NuwanS.
--
*Nuwan Silva*
*Senior Software Engineer - QA*
Mobile: +9477 980 4543
WSO2 Inc.
lean . enterprise . middlewear.
http://www.wso2.com
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
