Will apply and test in the setup. Regards, NuwanS.
On Mon, May 18, 2015 at 2:53 PM, Manoj Kumara <[email protected]> wrote: > Hi Nuwan, > > Please try out with attached patch. This was added on latest kernel > patch0010 as well. > > WSO2-CARBON-PATCH-4.2.0-1297.zip > <https://docs.google.com/a/wso2.com/file/d/0B-yMpNmsyVchNDZDLWVEVFl4VlU/edit?usp=drive_web> > > Regards, > Manoj > > > *Manoj Kumara* > Software Engineer > WSO2 Inc. http://wso2.com/ > *lean.enterprise.middleware* > Mobile: +94713448188 > > On Mon, May 18, 2015 at 2:48 PM, Afkham Azeez <[email protected]> wrote: > >> Please upgrade to Hazelcast 3.4.2 >> >> On Mon, May 18, 2015 at 9:12 AM, Nuwan Silva <[email protected]> wrote: >> >>> yes, this is continuously reproducible. Anyway will check the configs >>> again and verify. >>> >>> Regards, >>> NuwanS. >>> >>> On Sat, May 16, 2015 at 2:34 PM, Dinusha Senanayaka <[email protected]> >>> wrote: >>> >>>> HI Nuwan/ Asanthi, >>>> >>>> We configured 3-nodes cluster setup to test this scenario. Seems >>>> everything works fine. "Member left" notification propagate to all other >>>> nodes correctly. Also dep-sync works without issue with member left and >>>> joining. >>>> >>>> Anyway, is this issue continuously re-producible in your setup ? Could >>>> see some mismatch in the node that left and node that dep-sync has failed >>>> to connect in your logs. >>>> >>>> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: /192.168.48.2:4001 >>>> Packet not sent to -> Address[192.168.48.5]:4000 >>>> >>>> May be some network issue in the machine caused this issue ? >>>> >>>> Btw, we don't need to add dep-sync for store/publisher. Only to GW >>>> cluster is enough. >>>> >>>> Regards, >>>> Dinusha. >>>> >>>> >>>> On Thu, May 14, 2015 at 5:31 PM, Asanthi Kulasinghe <[email protected]> >>>> wrote: >>>> >>>>> Looping in Azeez and Dinusha >>>>> >>>>> >>>>> On Wed, May 13, 2015 at 2:33 PM, Nuwan Silva <[email protected]> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I came across the following issue where we see a depsync error when a >>>>>> single node in the cluster goes down and the other nodes not knowing it >>>>>> till its restarted. >>>>>> observation was that when a node (node 1) in the cluster goes down >>>>>> the other nodes (node 2 and node 3) do not know that node (node 1) is >>>>>> down. >>>>>> So it tries to synchronize its artifacts. >>>>>> >>>>>> further we observed that when the node which was down (node 1) >>>>>> restarts. the other nodes find out that node1 was down. >>>>>> >>>>>> TID: [0] [AM] [2015-05-13 09:00:08,861] INFO >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> - >>>>>> Member left [cc9950e3-af57-48b1-9f45-b511f0f30863]: / >>>>>> 192.168.48.2:4001 >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> TID: [0] [AM] [2015-05-13 09:00:08,877] INFO >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> - >>>>>> WKA member Host:192.168.48.2, Remote Host:null, Port: 4001, HTTP:8281, >>>>>> HTTPS:8244, Domain: wso2.am.storepub.domain, Sub-domain:worker, >>>>>> Active:true >>>>>> left cluster. >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> TID: [0] [AM] [2015-05-13 09:00:08,890] INFO >>>>>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - >>>>>> Elected this member [10bb1dec-4d25-4966-a39a-eb42e42d1706] as the >>>>>> Coordinator for the cluster [wso2.am.storepub.domain] >>>>>> {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} >>>>>> TID: [0] [AM] [2015-05-13 09:00:14,849] INFO >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> - >>>>>> Member joined [bc438078-ee97-4643-aadb-cf7ba970dc7e]: / >>>>>> 192.168.48.2:4001 >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> >>>>>> >>>>>> >>>>>> *Depsync Error:* >>>>>> >>>>>> TID: [0] [AM] [2015-05-13 08:52:03,333] ERROR >>>>>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - >>>>>> Deployment synchronization commit for tenant -1234 failed >>>>>> {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} >>>>>> com.hazelcast.core.HazelcastException: >>>>>> com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> >>>>>> Address[192.168.48.5]:4000 >>>>>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:45) >>>>>> at com.hazelcast.util.ExceptionUtil.rethrow(ExceptionUtil.java:40) >>>>>> at >>>>>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:371) >>>>>> at >>>>>> com.hazelcast.map.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:215) >>>>>> at >>>>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.containsKey(HazelcastDistributedMapProvider.java:119) >>>>>> at >>>>>> org.wso2.carbon.caching.impl.CacheImpl.containsKey(CacheImpl.java:237) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:291) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757) >>>>>> at >>>>>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262) >>>>>> at >>>>>> org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108) >>>>>> at >>>>>> org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96) >>>>>> at >>>>>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207) >>>>>> at >>>>>> org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: com.hazelcast.spi.exception.RetryableIOException: Packet >>>>>> not sent to -> Address[192.168.48.5]:4000 >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:360) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation.access$900(BasicInvocation.java:62) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation$ReInvocationTask.run(BasicInvocation.java:549) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> at >>>>>> com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) >>>>>> at ------ End remote and begin local stack-trace ------.(Unknown >>>>>> Source) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698) >>>>>> at >>>>>> com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676) >>>>>> at >>>>>> com.hazelcast.map.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:369) >>>>>> ... 22 more >>>>>> >>>>>> >>>>>> Regards, >>>>>> NuwanS. >>>>>> -- >>>>>> >>>>>> >>>>>> *Nuwan Silva* >>>>>> *Senior Software Engineer - QA* >>>>>> Mobile: +9477 980 4543 >>>>>> >>>>>> WSO2 Inc. >>>>>> lean . enterprise . middlewear. >>>>>> http://www.wso2.com >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Asanthi Kulasinghe* >>>>> WSO2 Inc; http://www.wso2.com/. >>>>> Mobile: +94777355522 >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dinusha Dilrukshi >>>> Associate Technical Lead >>>> WSO2 Inc.: http://wso2.com/ >>>> Mobile: +94725255071 >>>> Blog: http://dinushasblog.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> >>> >>> *Nuwan Silva* >>> *Senior Software Engineer - QA* >>> Mobile: +9477 980 4543 >>> >>> WSO2 Inc. >>> lean . enterprise . middlewear. >>> http://www.wso2.com >>> >> >> >> >> -- >> *Afkham Azeez* >> Director of Architecture; WSO2, Inc.; http://wso2.com >> Member; Apache Software Foundation; http://www.apache.org/ >> * <http://www.apache.org/>* >> *email: **[email protected]* <[email protected]> >> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >> *http://blog.afkham.org* <http://blog.afkham.org> >> *twitter: **http://twitter.com/afkham_azeez* >> <http://twitter.com/afkham_azeez> >> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >> <http://lk.linkedin.com/in/afkhamazeez>* >> >> *Lean . Enterprise . Middleware* >> > > -- *Nuwan Silva* *Senior Software Engineer - QA* Mobile: +9477 980 4543 WSO2 Inc. lean . enterprise . middlewear. http://www.wso2.com
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
