Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Maninda Edirisooriya]

We have not tested AWS clustering with old kernel. We moved to AWS
clustering with new kernel. I thought it was due to new kernel version as I
have heard AWS was working properly with products released with older
kernel.
Here I have started each node one after the other. And once a member was
stopped also any message was not printed.


On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne  wrote:

> Member joined message may not come with AWS membership scheme if all the
> members were started at the same time. However the member left message
> should appear if a member is stopped afterwards.
>
>
> On Monday, October 26, 2015, Maninda Edirisooriya 
> wrote:
>
>> I have setup the AWS clustering according to [1] but it seems the 
>> "AWSBasedMembershipScheme
>> Member joined" message is not displayed once a new member is joined and
>> no message when it left. This seems to be an issue with new kernel version.
>>
>> > class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
>> enable="true">
>>
>>
>> aws
>>
>>
>> 5701
>>
>>
>> xxx
>> zzz
>> sg-16f4b571
>> us-east-1
>> a_tag_key
>> a_tag_value
>>
>>
>> [1]
>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>
>>
>> Thanks.
>>
>>
>> *Maninda Edirisooriya*
>> Senior Software Engineer
>>
>> *WSO2, Inc.*lean.enterprise.middleware.
>>
>> *Blog* : http://maninda.blogspot.com/
>> *E-mail* : mani...@wso2.com
>> *Skype* : @manindae
>> *Twitter* : @maninda
>>
>> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:
>>
>>> In EC2 with AWS mode, you should use ports 5701 to 5800
>>>
>>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera 
>>> wrote:
>>>
 Hi Thusitha,

 4000 is the local host port specified in the axis2.xml

 quote axis2.xml
 "

 10.100.5.109

 
 

 
 4000"

 in DAS we are using wka as the membership scheme.

 cheers

 On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
 thusit...@wso2.com> wrote:

> Hi Anjana,
>
> I noticed following error line on the error stack that you posted.
>
> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>> this:Address[172.18.1.227]:4000
>
>
> Here it tries to connect using the port 4000. But AFAIK we usually use
> ports between 5701 to 5800. Did you set the port to 4000 in any instance?
>
> Thanks
> Thusitha
>
> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
> wrote:
>
>> Hi,
>>
>> So I'd a chat with Azeez, and the first thing he mentioned was, we
>> should be using the AWS mode in EC2 it seems, the docs can be found for
>> this here [1]. And his comments are, these kind of issues can be seen
>> sometimes there, but if it recovers eventually it would be okay, but if 
>> it
>> continues to give same kind of errors always, that would be a problem.
>>
>> [1]
>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>
>> Cheers,
>> Anjana.
>>
>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
>>> has reported some Hazlecast issues, specifically [1] and [2]. Some of 
>>> the
>>> errors as follows:-
>>>
>>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>>> Unable to test existence of resource
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
>>> partitionId: 5, replicaIndex: 0, operation:
>>> com.hazelcast.map.impl.operation.GetOperation, service: 
>>> hz:impl:mapService
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>>> at -- End remote and begin 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Maninda Edirisooriya]

And also I have tested the cluster with WKA membership scheme with old
kernel previously and it worked fine. But with new kernel version we got
some issues and that is why we had to move to AWS clustering mode.


On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
wrote:

> We have not tested AWS clustering with old kernel. We moved to AWS
> clustering with new kernel. I thought it was due to new kernel version as I
> have heard AWS was working properly with products released with older
> kernel.
> Here I have started each node one after the other. And once a member was
> stopped also any message was not printed.
>
>
>
> On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne  wrote:
>
>> Member joined message may not come with AWS membership scheme if all the
>> members were started at the same time. However the member left message
>> should appear if a member is stopped afterwards.
>>
>>
>> On Monday, October 26, 2015, Maninda Edirisooriya 
>> wrote:
>>
>>> I have setup the AWS clustering according to [1] but it seems the 
>>> "AWSBasedMembershipScheme
>>> Member joined" message is not displayed once a new member is joined and
>>> no message when it left. This seems to be an issue with new kernel version.
>>>
>>> >> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
>>> enable="true">
>>>
>>>
>>> aws
>>>
>>>
>>> 5701
>>>
>>>
>>> xxx
>>> zzz
>>> sg-16f4b571
>>> us-east-1
>>> a_tag_key
>>> a_tag_value
>>>
>>>
>>> [1]
>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>>
>>>
>>> Thanks.
>>>
>>>
>>> *Maninda Edirisooriya*
>>> Senior Software Engineer
>>>
>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>
>>> *Blog* : http://maninda.blogspot.com/
>>> *E-mail* : mani...@wso2.com
>>> *Skype* : @manindae
>>> *Twitter* : @maninda
>>>
>>> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:
>>>
 In EC2 with AWS mode, you should use ports 5701 to 5800

 On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera 
 wrote:

> Hi Thusitha,
>
> 4000 is the local host port specified in the axis2.xml
>
> quote axis2.xml
> "
>
> 10.100.5.109
>
> 
> 
>
> 
> 4000"
>
> in DAS we are using wka as the membership scheme.
>
> cheers
>
> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
> thusit...@wso2.com> wrote:
>
>> Hi Anjana,
>>
>> I noticed following error line on the error stack that you posted.
>>
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>>> this:Address[172.18.1.227]:4000
>>
>>
>> Here it tries to connect using the port 4000. But AFAIK we usually
>> use ports between 5701 to 5800. Did you set the port to 4000 in any
>> instance?
>>
>> Thanks
>> Thusitha
>>
>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
>> wrote:
>>
>>> Hi,
>>>
>>> So I'd a chat with Azeez, and the first thing he mentioned was, we
>>> should be using the AWS mode in EC2 it seems, the docs can be found for
>>> this here [1]. And his comments are, these kind of issues can be seen
>>> sometimes there, but if it recovers eventually it would be okay, but if 
>>> it
>>> continues to give same kind of errors always, that would be a problem.
>>>
>>> [1]
>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi everyone,

 When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
 has reported some Hazlecast issues, specifically [1] and [2]. Some of 
 the
 errors as follows:-

 TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
 Unable to test existence of resource
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
 this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
 partitionId: 5, replicaIndex: 0, operation:
 com.hazelcast.map.impl.operation.GetOperation, service: 
 hz:impl:mapService
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
 at
 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Imesh Gunaratne]

On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
 wrote:

>
> Here I have started each node one after the other. And once a member was
> stopped also any message was not printed.
>

Then it might be a connection issue between nodes.

On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
wrote:

> We have not tested AWS clustering with old kernel. We moved to AWS
> clustering with new kernel. I thought it was due to new kernel version as I
> have heard AWS was working properly with products released with older
> kernel.
> Here I have started each node one after the other. And once a member was
> stopped also any message was not printed.
>
>
>
> On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne  wrote:
>
>> Member joined message may not come with AWS membership scheme if all the
>> members were started at the same time. However the member left message
>> should appear if a member is stopped afterwards.
>>
>>
>> On Monday, October 26, 2015, Maninda Edirisooriya 
>> wrote:
>>
>>> I have setup the AWS clustering according to [1] but it seems the 
>>> "AWSBasedMembershipScheme
>>> Member joined" message is not displayed once a new member is joined and
>>> no message when it left. This seems to be an issue with new kernel version.
>>>
>>> >> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
>>> enable="true">
>>>
>>>
>>> aws
>>>
>>>
>>> 5701
>>>
>>>
>>> xxx
>>> zzz
>>> sg-16f4b571
>>> us-east-1
>>> a_tag_key
>>> a_tag_value
>>>
>>>
>>> [1]
>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>>
>>>
>>> Thanks.
>>>
>>>
>>> *Maninda Edirisooriya*
>>> Senior Software Engineer
>>>
>>> *WSO2, Inc.*lean.enterprise.middleware.
>>>
>>> *Blog* : http://maninda.blogspot.com/
>>> *E-mail* : mani...@wso2.com
>>> *Skype* : @manindae
>>> *Twitter* : @maninda
>>>
>>> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:
>>>
 In EC2 with AWS mode, you should use ports 5701 to 5800

 On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera 
 wrote:

> Hi Thusitha,
>
> 4000 is the local host port specified in the axis2.xml
>
> quote axis2.xml
> "
>
> 10.100.5.109
>
> 
> 
>
> 
> 4000"
>
> in DAS we are using wka as the membership scheme.
>
> cheers
>
> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
> thusit...@wso2.com> wrote:
>
>> Hi Anjana,
>>
>> I noticed following error line on the error stack that you posted.
>>
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>>> this:Address[172.18.1.227]:4000
>>
>>
>> Here it tries to connect using the port 4000. But AFAIK we usually
>> use ports between 5701 to 5800. Did you set the port to 4000 in any
>> instance?
>>
>> Thanks
>> Thusitha
>>
>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
>> wrote:
>>
>>> Hi,
>>>
>>> So I'd a chat with Azeez, and the first thing he mentioned was, we
>>> should be using the AWS mode in EC2 it seems, the docs can be found for
>>> this here [1]. And his comments are, these kind of issues can be seen
>>> sometimes there, but if it recovers eventually it would be okay, but if 
>>> it
>>> continues to give same kind of errors always, that would be a problem.
>>>
>>> [1]
>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi everyone,

 When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
 has reported some Hazlecast issues, specifically [1] and [2]. Some of 
 the
 errors as follows:-

 TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
 Unable to test existence of resource
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
 this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
 partitionId: 5, replicaIndex: 0, operation:
 com.hazelcast.map.impl.operation.GetOperation, service: 
 hz:impl:mapService
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
 at
 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Maninda Edirisooriya]

No. It is because the cluster was not properly set. I could access other
nodes from these nodes. Even though the cluster was setup it seems nodes
does not recognize other nodes as members.


On Tue, Oct 27, 2015 at 2:02 PM, Imesh Gunaratne  wrote:

> On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
>  wrote:
>
>>
>> Here I have started each node one after the other. And once a member was
>> stopped also any message was not printed.
>>
>
> Then it might be a connection issue between nodes.
>
> On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
> wrote:
>
>> We have not tested AWS clustering with old kernel. We moved to AWS
>> clustering with new kernel. I thought it was due to new kernel version as I
>> have heard AWS was working properly with products released with older
>> kernel.
>> Here I have started each node one after the other. And once a member was
>> stopped also any message was not printed.
>>
>>
>>
>> On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne  wrote:
>>
>>> Member joined message may not come with AWS membership scheme if all the
>>> members were started at the same time. However the member left message
>>> should appear if a member is stopped afterwards.
>>>
>>>
>>> On Monday, October 26, 2015, Maninda Edirisooriya 
>>> wrote:
>>>
 I have setup the AWS clustering according to [1] but it seems the 
 "AWSBasedMembershipScheme
 Member joined" message is not displayed once a new member is joined
 and no message when it left. This seems to be an issue with new kernel
 version.

 >>> class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
 enable="true">


 aws


 5701


 xxx
 zzz
 sg-16f4b571
 us-east-1
 a_tag_key
 a_tag_value


 [1]
 https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode


 Thanks.


 *Maninda Edirisooriya*
 Senior Software Engineer

 *WSO2, Inc.*lean.enterprise.middleware.

 *Blog* : http://maninda.blogspot.com/
 *E-mail* : mani...@wso2.com
 *Skype* : @manindae
 *Twitter* : @maninda

 On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:

> In EC2 with AWS mode, you should use ports 5701 to 5800
>
> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera 
> wrote:
>
>> Hi Thusitha,
>>
>> 4000 is the local host port specified in the axis2.xml
>>
>> quote axis2.xml
>> "
>>
>> 10.100.5.109
>>
>> 
>> 
>>
>> 
>> 4000"
>>
>> in DAS we are using wka as the membership scheme.
>>
>> cheers
>>
>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
>> thusit...@wso2.com> wrote:
>>
>>> Hi Anjana,
>>>
>>> I noticed following error line on the error stack that you posted.
>>>
>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
 this:Address[172.18.1.227]:4000
>>>
>>>
>>> Here it tries to connect using the port 4000. But AFAIK we usually
>>> use ports between 5701 to 5800. Did you set the port to 4000 in any
>>> instance?
>>>
>>> Thanks
>>> Thusitha
>>>
>>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi,

 So I'd a chat with Azeez, and the first thing he mentioned was, we
 should be using the AWS mode in EC2 it seems, the docs can be found for
 this here [1]. And his comments are, these kind of issues can be seen
 sometimes there, but if it recovers eventually it would be okay, but 
 if it
 continues to give same kind of errors always, that would be a problem.

 [1]
 https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode

 Cheers,
 Anjana.

 On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
 wrote:

> Hi everyone,
>
> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
> has reported some Hazlecast issues, specifically [1] and [2]. Some of 
> the
> errors as follows:-
>
> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
> Unable to test existence of resource
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
> partitionId: 5, replicaIndex: 0, operation:
> com.hazelcast.map.impl.operation.GetOperation, service: 
> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Maninda Edirisooriya]

I setup two Carbon 4.4.2 servers in two EC2 nodes and added the above
mentioned configurations in the axis2.xml and started (one after the other)
and the same issue was observed. No message was logged once one server was
shut down in the other node backend logs.

Thanks.


*Maninda Edirisooriya*
Senior Software Engineer

*WSO2, Inc.*lean.enterprise.middleware.

*Blog* : http://maninda.blogspot.com/
*E-mail* : mani...@wso2.com
*Skype* : @manindae
*Twitter* : @maninda

On Tue, Oct 27, 2015 at 4:33 PM, Maninda Edirisooriya 
wrote:

> No. It is because the cluster was not properly set. I could access other
> nodes from these nodes. Even though the cluster was setup it seems nodes
> does not recognize other nodes as members.
>
>
>
> On Tue, Oct 27, 2015 at 2:02 PM, Imesh Gunaratne  wrote:
>
>> On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
>>  wrote:
>>
>>>
>>> Here I have started each node one after the other. And once a member was
>>> stopped also any message was not printed.
>>>
>>
>> Then it might be a connection issue between nodes.
>>
>> On Tue, Oct 27, 2015 at 12:55 PM, Maninda Edirisooriya 
>> wrote:
>>
>>> We have not tested AWS clustering with old kernel. We moved to AWS
>>> clustering with new kernel. I thought it was due to new kernel version as I
>>> have heard AWS was working properly with products released with older
>>> kernel.
>>> Here I have started each node one after the other. And once a member was
>>> stopped also any message was not printed.
>>>
>>>
>>>
>>> On Mon, Oct 26, 2015 at 11:26 PM, Imesh Gunaratne 
>>> wrote:
>>>
 Member joined message may not come with AWS membership scheme if all
 the members were started at the same time. However the member left message
 should appear if a member is stopped afterwards.


 On Monday, October 26, 2015, Maninda Edirisooriya 
 wrote:

> I have setup the AWS clustering according to [1] but it seems the 
> "AWSBasedMembershipScheme
> Member joined" message is not displayed once a new member is joined
> and no message when it left. This seems to be an issue with new kernel
> version.
>
>  class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
> enable="true">
>
>
> aws
>
>
> 5701
>
>
> xxx
> zzz
> sg-16f4b571
> us-east-1
> a_tag_key
> a_tag_value
>
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
>
> Thanks.
>
>
> *Maninda Edirisooriya*
> Senior Software Engineer
>
> *WSO2, Inc.*lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *E-mail* : mani...@wso2.com
> *Skype* : @manindae
> *Twitter* : @maninda
>
> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:
>
>> In EC2 with AWS mode, you should use ports 5701 to 5800
>>
>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera 
>> wrote:
>>
>>> Hi Thusitha,
>>>
>>> 4000 is the local host port specified in the axis2.xml
>>>
>>> quote axis2.xml
>>> "
>>>
>>> 10.100.5.109
>>>
>>> 
>>> 
>>>
>>> 
>>> 4000"
>>>
>>> in DAS we are using wka as the membership scheme.
>>>
>>> cheers
>>>
>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
>>> thusit...@wso2.com> wrote:
>>>
 Hi Anjana,

 I noticed following error line on the error stack that you posted.

 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000


 Here it tries to connect using the port 4000. But AFAIK we usually
 use ports between 5701 to 5800. Did you set the port to 4000 in any
 instance?

 Thanks
 Thusitha

 On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
 wrote:

> Hi,
>
> So I'd a chat with Azeez, and the first thing he mentioned was, we
> should be using the AWS mode in EC2 it seems, the docs can be found 
> for
> this here [1]. And his comments are, these kind of issues can be seen
> sometimes there, but if it recovers eventually it would be okay, but 
> if it
> continues to give same kind of errors always, that would be a problem.
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
> Cheers,
> Anjana.
>
> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
> wrote:
>
>> Hi everyone,
>>
>> When testing DAS in EC2, and also 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Imesh Gunaratne]

Member joined message may not come with AWS membership scheme if all the
members were started at the same time. However the member left message
should appear if a member is stopped afterwards.

On Monday, October 26, 2015, Maninda Edirisooriya  wrote:

> I have setup the AWS clustering according to [1] but it seems the 
> "AWSBasedMembershipScheme
> Member joined" message is not displayed once a new member is joined and
> no message when it left. This seems to be an issue with new kernel version.
>
>  class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
> enable="true">
>
>
> aws
>
>
> 5701
>
>
> xxx
> zzz
> sg-16f4b571
> us-east-1
> a_tag_key
> a_tag_value
>
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
>
> Thanks.
>
>
> *Maninda Edirisooriya*
> Senior Software Engineer
>
> *WSO2, Inc.*lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *E-mail* : mani...@wso2.com
> 
> *Skype* : @manindae
> *Twitter* : @maninda
>
> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  > wrote:
>
>> In EC2 with AWS mode, you should use ports 5701 to 5800
>>
>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera > > wrote:
>>
>>> Hi Thusitha,
>>>
>>> 4000 is the local host port specified in the axis2.xml
>>>
>>> quote axis2.xml
>>> "
>>>
>>> 10.100.5.109
>>>
>>> 
>>> 
>>>
>>> 
>>> 4000"
>>>
>>> in DAS we are using wka as the membership scheme.
>>>
>>> cheers
>>>
>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
>>> thusit...@wso2.com >
>>> wrote:
>>>
 Hi Anjana,

 I noticed following error line on the error stack that you posted.

 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000


 Here it tries to connect using the port 4000. But AFAIK we usually use
 ports between 5701 to 5800. Did you set the port to 4000 in any instance?

 Thanks
 Thusitha

 On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando > wrote:

> Hi,
>
> So I'd a chat with Azeez, and the first thing he mentioned was, we
> should be using the AWS mode in EC2 it seems, the docs can be found for
> this here [1]. And his comments are, these kind of issues can be seen
> sometimes there, but if it recovers eventually it would be okay, but if it
> continues to give same kind of errors always, that would be a problem.
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
> Cheers,
> Anjana.
>
> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando  > wrote:
>
>> Hi everyone,
>>
>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
>> has reported some Hazlecast issues, specifically [1] and [2]. Some of the
>> errors as follows:-
>>
>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>> Unable to test existence of resource
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
>> partitionId: 5, replicaIndex: 0, operation:
>> com.hazelcast.map.impl.operation.GetOperation, service: 
>> hz:impl:mapService
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>> at -- End remote and begin local stack-trace --.(Unknown
>> Source)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
>> at
>> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Afkham Azeez]

Are the same settings working with the old kernel?

On Mon, Oct 26, 2015 at 10:21 PM, Maninda Edirisooriya 
wrote:

> I have setup the AWS clustering according to [1] but it seems the 
> "AWSBasedMembershipScheme
> Member joined" message is not displayed once a new member is joined and
> no message when it left. This seems to be an issue with new kernel version.
>
>  class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
> enable="true">
>
>
> aws
>
>
> 5701
>
>
> xxx
> zzz
> sg-16f4b571
> us-east-1
> a_tag_key
> a_tag_value
>
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
>
> Thanks.
>
>
> *Maninda Edirisooriya*
> Senior Software Engineer
>
> *WSO2, Inc.*lean.enterprise.middleware.
>
> *Blog* : http://maninda.blogspot.com/
> *E-mail* : mani...@wso2.com
> *Skype* : @manindae
> *Twitter* : @maninda
>
> On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:
>
>> In EC2 with AWS mode, you should use ports 5701 to 5800
>>
>> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera  wrote:
>>
>>> Hi Thusitha,
>>>
>>> 4000 is the local host port specified in the axis2.xml
>>>
>>> quote axis2.xml
>>> "
>>>
>>> 10.100.5.109
>>>
>>> 
>>> 
>>>
>>> 
>>> 4000"
>>>
>>> in DAS we are using wka as the membership scheme.
>>>
>>> cheers
>>>
>>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
>>> thusit...@wso2.com> wrote:
>>>
 Hi Anjana,

 I noticed following error line on the error stack that you posted.

 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000


 Here it tries to connect using the port 4000. But AFAIK we usually use
 ports between 5701 to 5800. Did you set the port to 4000 in any instance?

 Thanks
 Thusitha

 On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
 wrote:

> Hi,
>
> So I'd a chat with Azeez, and the first thing he mentioned was, we
> should be using the AWS mode in EC2 it seems, the docs can be found for
> this here [1]. And his comments are, these kind of issues can be seen
> sometimes there, but if it recovers eventually it would be okay, but if it
> continues to give same kind of errors always, that would be a problem.
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
> Cheers,
> Anjana.
>
> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
> wrote:
>
>> Hi everyone,
>>
>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA
>> has reported some Hazlecast issues, specifically [1] and [2]. Some of the
>> errors as follows:-
>>
>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>> Unable to test existence of resource
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
>> partitionId: 5, replicaIndex: 0, operation:
>> com.hazelcast.map.impl.operation.GetOperation, service: 
>> hz:impl:mapService
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>> at -- End remote and begin local stack-trace --.(Unknown
>> Source)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
>> at
>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
>> at
>> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Maninda Edirisooriya]

I have setup the AWS clustering according to [1] but it seems the
"AWSBasedMembershipScheme
Member joined" message is not displayed once a new member is joined and no
message when it left. This seems to be an issue with new kernel version.




aws


5701


xxx
zzz
sg-16f4b571
us-east-1
a_tag_key
a_tag_value


[1]
https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode


Thanks.


*Maninda Edirisooriya*
Senior Software Engineer

*WSO2, Inc.*lean.enterprise.middleware.

*Blog* : http://maninda.blogspot.com/
*E-mail* : mani...@wso2.com
*Skype* : @manindae
*Twitter* : @maninda

On Sun, Oct 25, 2015 at 6:59 PM, Afkham Azeez  wrote:

> In EC2 with AWS mode, you should use ports 5701 to 5800
>
> On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera  wrote:
>
>> Hi Thusitha,
>>
>> 4000 is the local host port specified in the axis2.xml
>>
>> quote axis2.xml
>> "
>>
>> 10.100.5.109
>>
>> 
>> 
>>
>> 
>> 4000"
>>
>> in DAS we are using wka as the membership scheme.
>>
>> cheers
>>
>> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
>> thusit...@wso2.com> wrote:
>>
>>> Hi Anjana,
>>>
>>> I noticed following error line on the error stack that you posted.
>>>
>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
 this:Address[172.18.1.227]:4000
>>>
>>>
>>> Here it tries to connect using the port 4000. But AFAIK we usually use
>>> ports between 5701 to 5800. Did you set the port to 4000 in any instance?
>>>
>>> Thanks
>>> Thusitha
>>>
>>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi,

 So I'd a chat with Azeez, and the first thing he mentioned was, we
 should be using the AWS mode in EC2 it seems, the docs can be found for
 this here [1]. And his comments are, these kind of issues can be seen
 sometimes there, but if it recovers eventually it would be okay, but if it
 continues to give same kind of errors always, that would be a problem.

 [1]
 https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode

 Cheers,
 Anjana.

 On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
 wrote:

> Hi everyone,
>
> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
> reported some Hazlecast issues, specifically [1] and [2]. Some of the
> errors as follows:-
>
> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
> Unable to test existence of resource
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
> partitionId: 5, replicaIndex: 0, operation:
> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
> at -- End remote and begin local stack-trace --.(Unknown
> Source)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
> at
> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
> at
> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
> at
> com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
> at
> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
> at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)
>
> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Afkham Azeez]

In EC2 with AWS mode, you should use ports 5701 to 5800

On Sun, Oct 25, 2015 at 5:52 PM, Niranda Perera  wrote:

> Hi Thusitha,
>
> 4000 is the local host port specified in the axis2.xml
>
> quote axis2.xml
> "
>
> 10.100.5.109
>
> 
> 
>
> 
> 4000"
>
> in DAS we are using wka as the membership scheme.
>
> cheers
>
> On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
> thusit...@wso2.com> wrote:
>
>> Hi Anjana,
>>
>> I noticed following error line on the error stack that you posted.
>>
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>>> this:Address[172.18.1.227]:4000
>>
>>
>> Here it tries to connect using the port 4000. But AFAIK we usually use
>> ports between 5701 to 5800. Did you set the port to 4000 in any instance?
>>
>> Thanks
>> Thusitha
>>
>> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando  wrote:
>>
>>> Hi,
>>>
>>> So I'd a chat with Azeez, and the first thing he mentioned was, we
>>> should be using the AWS mode in EC2 it seems, the docs can be found for
>>> this here [1]. And his comments are, these kind of issues can be seen
>>> sometimes there, but if it recovers eventually it would be okay, but if it
>>> continues to give same kind of errors always, that would be a problem.
>>>
>>> [1]
>>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>>
>>> Cheers,
>>> Anjana.
>>>
>>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando 
>>> wrote:
>>>
 Hi everyone,

 When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
 reported some Hazlecast issues, specifically [1] and [2]. Some of the
 errors as follows:-

 TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
 Unable to test existence of resource
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
 com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
 this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
 partitionId: 5, replicaIndex: 0, operation:
 com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
 at
 com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
 at
 com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
 at
 com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
 at
 com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
 at -- End remote and begin local stack-trace --.(Unknown
 Source)
 at
 com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
 at
 com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
 at
 com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
 at
 com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
 at
 com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
 at
 com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
 at
 com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
 at
 org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
 at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)

 


 TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
 Unable to test existence of resource
 {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
 com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
 caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
 com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
 at
 com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
 at
 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Thusitha Thilina Dayaratne]

Hi Anjana,

I noticed following error line on the error stack that you posted.

com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000


Here it tries to connect using the port 4000. But AFAIK we usually use
ports between 5701 to 5800. Did you set the port to 4000 in any instance?

Thanks
Thusitha

On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando  wrote:

> Hi,
>
> So I'd a chat with Azeez, and the first thing he mentioned was, we should
> be using the AWS mode in EC2 it seems, the docs can be found for this here
> [1]. And his comments are, these kind of issues can be seen sometimes
> there, but if it recovers eventually it would be okay, but if it continues
> to give same kind of errors always, that would be a problem.
>
> [1]
> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>
> Cheers,
> Anjana.
>
> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando  wrote:
>
>> Hi everyone,
>>
>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
>> reported some Hazlecast issues, specifically [1] and [2]. Some of the
>> errors as follows:-
>>
>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>> Unable to test existence of resource
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
>> partitionId: 5, replicaIndex: 0, operation:
>> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>> at -- End remote and begin local stack-trace --.(Unknown
>> Source)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
>> at
>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
>> at
>> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
>> at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
>> at
>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
>> at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)
>>
>> 
>>
>>
>> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>> Unable to test existence of resource
>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>> com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
>> caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
>> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
>> at
>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>> at
>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>>
>> 
>>
>>  ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} -
>> [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response:
>> JoinRequest{packetVersion=4, buildNumber=20150826,
>> address=Address[172.18.1.229]:4000,
>> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
>> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
>> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Niranda Perera]

Hi Thusitha,

4000 is the local host port specified in the axis2.xml

quote axis2.xml
"

10.100.5.109





4000"

in DAS we are using wka as the membership scheme.

cheers

On Sun, Oct 25, 2015 at 4:47 PM, Thusitha Thilina Dayaratne <
thusit...@wso2.com> wrote:

> Hi Anjana,
>
> I noticed following error line on the error stack that you posted.
>
> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>> this:Address[172.18.1.227]:4000
>
>
> Here it tries to connect using the port 4000. But AFAIK we usually use
> ports between 5701 to 5800. Did you set the port to 4000 in any instance?
>
> Thanks
> Thusitha
>
> On Sat, Oct 24, 2015 at 5:27 PM, Anjana Fernando  wrote:
>
>> Hi,
>>
>> So I'd a chat with Azeez, and the first thing he mentioned was, we should
>> be using the AWS mode in EC2 it seems, the docs can be found for this here
>> [1]. And his comments are, these kind of issues can be seen sometimes
>> there, but if it recovers eventually it would be okay, but if it continues
>> to give same kind of errors always, that would be a problem.
>>
>> [1]
>> https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode
>>
>> Cheers,
>> Anjana.
>>
>> On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando  wrote:
>>
>>> Hi everyone,
>>>
>>> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
>>> reported some Hazlecast issues, specifically [1] and [2]. Some of the
>>> errors as follows:-
>>>
>>> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>>> Unable to test existence of resource
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>>> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
>>> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
>>> partitionId: 5, replicaIndex: 0, operation:
>>> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
>>> at -- End remote and begin local stack-trace --.(Unknown
>>> Source)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
>>> at
>>> com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
>>> at
>>> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
>>> at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)
>>>
>>> 
>>>
>>>
>>> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
>>> Unable to test existence of resource
>>> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
>>> com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
>>> caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
>>> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
>>> at
>>> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
>>> at
>>> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>>>
>>> 

Re: [Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Anjana Fernando]

Hi,

So I'd a chat with Azeez, and the first thing he mentioned was, we should
be using the AWS mode in EC2 it seems, the docs can be found for this here
[1]. And his comments are, these kind of issues can be seen sometimes
there, but if it recovers eventually it would be okay, but if it continues
to give same kind of errors always, that would be a problem.

[1]
https://docs.wso2.com/display/CLUSTER420/Setting+up+a+Cluster+in+AWS+Mode

Cheers,
Anjana.

On Sat, Oct 24, 2015 at 5:11 PM, Anjana Fernando  wrote:

> Hi everyone,
>
> When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
> reported some Hazlecast issues, specifically [1] and [2]. Some of the
> errors as follows:-
>
> TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
> Unable to test existence of resource
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
> com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
> this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
> partitionId: 5, replicaIndex: 0, operation:
> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
> at -- End remote and begin local stack-trace --.(Unknown
> Source)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
> at
> com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
> at
> com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
> at
> com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
> at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
> at
> org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
> at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)
>
> 
>
>
> TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
> Unable to test existence of resource
> {org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
> com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
> caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
> com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
> at
> com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
> at
> com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
>
> 
>
>  ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} -
> [172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response:
> JoinRequest{packetVersion=4, buildNumber=20150826,
> address=Address[172.18.1.229]:4000,
> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
> {com.hazelcast.cluster.impl.operations.JoinCheckOperation}
> com.hazelcast.core.HazelcastException: Cannot send response:
> JoinRequest{packetVersion=4, buildNumber=20150826,
> address=Address[172.18.1.229]:4000,
> uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
> memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
> at
> com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131)
> at
> 

[Dev] Hazelcast Issues in EC2 (Kernel 4.4.2)

[Anjana Fernando]

Hi everyone,

When testing DAS in EC2, and also with the latest Kernel 4.4.2, QA has
reported some Hazlecast issues, specifically [1] and [2]. Some of the
errors as follows:-

TID: [-1234] [] [2015-10-24 08:40:05,130] ERROR
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
Unable to test existence of resource
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
com.hazelcast.spi.exception.WrongTargetException: WrongTarget!
this:Address[172.18.1.227]:4000, target:Address[172.18.1.228]:4000,
partitionId: 5, replicaIndex: 0, operation:
com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:315)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.doRun(OperationThread.java:101)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.run(OperationThread.java:76)
at -- End remote and begin local stack-trace --.(Unknown Source)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at
com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:456)
at
com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:275)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:82)
at
org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:141)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:203)




TID: [-1234] [] [2015-10-24 08:44:18,800] ERROR
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler} -
Unable to test existence of resource
{org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler}
com.hazelcast.spi.exception.CallerNotMemberException: Not Member!
caller:Address[172.18.1.228]:4000, partitionId: 30, operation:
com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureValidMember(OperationRunnerImpl.java:336)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:308)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.processPacket(OperationThread.java:142)
at
com.hazelcast.spi.impl.operationexecutor.classic.OperationThread.process(OperationThread.java:115)



 ERROR {com.hazelcast.cluster.impl.operations.JoinCheckOperation} -
[172.18.1.229]:4000 [wso2.qa.das.domain] [3.5.2] Cannot send response:
JoinRequest{packetVersion=4, buildNumber=20150826,
address=Address[172.18.1.229]:4000,
uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
{com.hazelcast.cluster.impl.operations.JoinCheckOperation}
com.hazelcast.core.HazelcastException: Cannot send response:
JoinRequest{packetVersion=4, buildNumber=20150826,
address=Address[172.18.1.229]:4000,
uuid='6848559f-141d-47cd-9e75-53bb2de10d52', credentials=null,
memberCount=4, tryCount=0} to Address[172.18.1.228]:4000
at
com.hazelcast.spi.impl.ResponseHandlerFactory$RemoteInvocationResponseHandler.sendResponse(ResponseHandlerFactory.java:131)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleResponse(OperationRunnerImpl.java:204)



Have any one of you guys seen issues similar to the above in EC2? .. and
what are the possibilities this could arise, the most obvious thing would
be networking issues that can arise in EC2, which maybe related to a higher
latency that's there or something. Any of you guys know how to maybe tune
HZ in this situations to have a higher timeout values or anything like
that, or else, doing this in EC2 is altogether not recommended? ..

[1] https://wso2.org/jira/browse/DAS-301
[2] https://wso2.org/jira/browse/DAS-302

Cheers,
Anjana.
-- 
*Anjana Fernando*
Senior Technical Lead