On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya <[email protected]> wrote:
> > Here I have started each node one after the other. And once a member was > stopped also any message was not printed. > Then it might be a connection issue between nodes. On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya <[email protected]> wrote: > We have not tested AWS clustering with old kernel. We moved to AWS > clustering with new kernel. I thought it was due to new kernel version as I > have heard AWS was working properly with products released with older > kernel. > Here I have started each node one after the other. And once a member was > stopped also any message was not printed. > > > > On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne <[email protected]> wrote: > >> Member joined message may not come with AWS membership scheme if all the >> members were started at the same time. However the member left message >> should appear if a member is stopped afterwards. >> >> >> On Monday, October 26, 2015, Maninda Edirisooriya <[email protected]> >> wrote: >> >>> I have setup the AWS clustering according to [1] but it seems the >>> "AWSBasedMembershipScheme >>> Member joined" message is not displayed once a new member is joined and >>> no message when it left. This seems to be an issue with new kernel version. >>> >>> <clustering >>> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" >>> enable="true"> >>> >>> >>> <parameter name="membershipScheme">aws</parameter> >>> >>> >>> <parameter name="localMemberPort">5701</parameter> >>> >>> >>> <parameter name="accessKey">xxxxxxx</parameter> >>> <parameter name="secretKey">zzzzzzz</parameter> >>> <parameter name="securityGroup">sg-16f4b571</parameter> >>> <parameter name="region">us-east-1</parameter> >>> <parameter name="tagKey">a_tag_key</parameter> >>> <parameter name="tagValue">a_tag_value</parameter> >>> >>> >>> [1] >>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >>> >>> >>> Thanks. >>> >>> >>> *Maninda Edirisooriya* >>> Senior Software Engineer >>> >>> *WSO2, Inc.*lean.enterprise.middleware. >>> >>> *Blog* : http://maninda.blogspot.com/ >>> *E-mail* : [email protected] >>> *Skype* : @manindae >>> *Twitter* : @maninda >>> >>> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez <[email protected]> wrote: >>> >>>> In EC2 with AWS mode, you should use ports 5701 to 5800 >>>> >>>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera <[email protected]> >>>> wrote: >>>> >>>>> Hi Thusitha, >>>>> >>>>> 4000 is the local host port specified in the axis2.xml >>>>> >>>>> quote axis2.xml >>>>> " <!-- The host name or IP address of this member --> >>>>> >>>>> <parameter name="localMemberHost">10.100.5.109</parameter> >>>>> >>>>> <!-- >>>>> The bind adress of this member. The difference between >>>>> localMemberHost & localMemberBindAddress >>>>> is that localMemberHost is the one that is advertised by >>>>> this member, while localMemberBindAddress >>>>> is the address to which this member is bound to. >>>>> --> >>>>> <!-- >>>>> <parameter >>>>> name="localMemberBindAddress">10.100.5.109</parameter> >>>>> --> >>>>> >>>>> <!-- >>>>> The TCP port used by this member. This is the port through >>>>> which other nodes will >>>>> contact this member >>>>> --> >>>>> <parameter name="localMemberPort">4000</parameter>" >>>>> >>>>> in DAS we are using wka as the membership scheme. >>>>> >>>>> cheers >>>>> >>>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> I noticed following error line on the error stack that you posted. >>>>>> >>>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>>> this:Address[172.18.1.227]:4000 >>>>>> >>>>>> >>>>>> Here it tries to connect using the port 4000. But AFAIK we usually >>>>>> use ports between 5701 to 5800. Did you set the port to 4000 in any >>>>>> instance? >>>>>> >>>>>> Thanks >>>>>> Thusitha >>>>>> >>>>>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> So I'd a chat with Azeez, and the first thing he mentioned was, we >>>>>>> should be using the AWS mode in EC2 it seems, the docs can be found for >>>>>>> this here [1]. And his comments are, these kind of issues can be seen >>>>>>> sometimes there, but if it recovers eventually it would be okay, but if >>>>>>> it >>>>>>> continues to give same kind of errors always, that would be a problem. >>>>>>> >>>>>>> [1] >>>>>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA >>>>>>>> has reported some Hazlecast issues, specifically [1] and [2]. Some of >>>>>>>> the >>>>>>>> errors as follows:- >>>>>>>> >>>>>>>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>>> Unable to test existence of resource >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget! >>>>>>>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000, >>>>>>>> partitionId: 5, replicaIndex: 0, operation: >>>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>>> hz:impl:mapService >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76) >>>>>>>> at ------ End remote and begin local stack-trace >>>>>>>> ------.(Unknown Source) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275) >>>>>>>> at >>>>>>>> com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141) >>>>>>>> at >>>>>>>> org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> >>>>>>>> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} - >>>>>>>> Unable to test existence of resource >>>>>>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} >>>>>>>> com.hazelcast.spi.exception.CallerNotMemberException: Not Member! >>>>>>>> caller:Address[172.18.1.228]:4000, partitionId: 30, operation: >>>>>>>> com.hazelcast.map.impl.operation.GetOperation, service: >>>>>>>> hz:impl:mapService >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} >>>>>>>> - [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send >>>>>>>> response: >>>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>>> address=Address[172.18.1.229]:4000, >>>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>>> {com.hazelcast.cluster.impl.operations.JoinCheckOperation} >>>>>>>> com.hazelcast.core.HazelcastException: Cannot send response: >>>>>>>> JoinRequest{packetVersion=4, buildNumber=20150826, >>>>>>>> address=Address[172.18.1.229]:4000, >>>>>>>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null, >>>>>>>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000 >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131) >>>>>>>> at >>>>>>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204) >>>>>>>> >>>>>>>> ==================================================================== >>>>>>>> >>>>>>>> Have any one of you guys seen issues similar to the above in EC2? >>>>>>>> .. and what are the possibilities this could arise, the most obvious >>>>>>>> thing >>>>>>>> would be networking issues that can arise in EC2, which maybe related >>>>>>>> to a >>>>>>>> higher latency that's there or something. Any of you guys know how to >>>>>>>> maybe >>>>>>>> tune HZ in this situations to have a higher timeout values or anything >>>>>>>> like >>>>>>>> that, or else, doing this in EC2 is altogether not recommended? .. >>>>>>>> >>>>>>>> [1] https://wso2.org/jira/browse/DAS-301 >>>>>>>> [2] https://wso2.org/jira/browse/DAS-302 >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thusitha Dayaratne >>>>>> Software Engineer >>>>>> WSO2 Inc. - lean . enterprise . middleware | wso2.com >>>>>> >>>>>> Mobile +94712756809 >>>>>> Blog alokayasoya.blogspot.com >>>>>> About http://about.me/thusithathilina >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Niranda Perera* >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94-71-554-8430 >>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>> https://pythagoreanscript.wordpress.com/ >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Afkham Azeez* >>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>> Member; Apache Software Foundation; http://www.apache.org/ >>>> * <http://www.apache.org/>* >>>> *email: **[email protected]* >>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>> *twitter: **http://twitter.com/afkham_azeez* >>>> <http://twitter.com/afkham_azeez> >>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>> >>>> *Lean . Enterprise . Middleware* >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >> >> -- >> *Imesh Gunaratne* >> Senior Technical Lead >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: http://imesh.gunaratne.org >> Lean . Enterprise . Middleware >> >> >> > -- *Imesh Gunaratne* Senior Technical Lead WSO2 Inc: http://wso2.com T: +94 11 214 5345 M: +94 77 374 2057 W: http://imesh.gunaratne.org Lean . Enterprise . Middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
