Gayashan, please share your latest findings on this. When we see the member left msg, the current member list is updated with that event (the member gets removed). So above can occur if that is not happening accordingly. We should also compare the same with and without hazelcast upgrade.
On Fri, Oct 31, 2014 at 5:30 PM, Gayashan Amarasinghe <[email protected]> wrote: > Hi all, > > For Carbon testing we have a worker-mgt cluster fronted by ELB and > requests keep coming in from a jmeter client. During this if one (or more) > of the worker nodes were shutdown, after some time the ELB stops sending > requests to the nodes and the connection times out. Following log gets > printed in the ELB. > > TID: [0] [ELB] [2014-10-31 06:27:32,517] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port: > 4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, > Active:true . Error Code: 101503 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:32,519] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Dropping the faulty/unreachable Member with Domain:wso2.as.domain, > Host:172.31.7.214, Port:4100 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:32,738] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:172.31.0.128, Remote Host:null, Port: 4100, HTTP:9763, > HTTPS:9443, Domain: wso2.as.domain, Sub-domain:worker, Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:32,740] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : /172.31.7.214:9765 > {org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2014-10-31 06:27:32,743] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port: > 4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, > Active:true . Error Code: 101503 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:32,745] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Dropping the faulty/unreachable Member with Domain:wso2.as.domain, > Host:172.31.7.214, Port:4100 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:33,518] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765, > HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:33,520] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : /172.31.7.214:9765 > {org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2014-10-31 06:27:33,523] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : /172.31.7.214:9765 > {org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2014-10-31 06:27:33,744] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765, > HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:33,745] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port: > 4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, > Active:true . Error Code: 101503 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:33,745] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : /172.31.7.214:9765 > {org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2014-10-31 06:27:33,747] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Dropping the faulty/unreachable Member with Domain:wso2.as.domain, > Host:172.31.7.214, Port:4100 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:34,746] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:172.31.0.128, Remote Host:null, Port: 4100, HTTP:9763, > HTTPS:9443, Domain: wso2.as.domain, Sub-domain:worker, Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:34,748] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port: > 4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, > Active:true . Error Code: 101503 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:34,750] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Dropping the faulty/unreachable Member with Domain:wso2.as.domain, > Host:172.31.7.214, Port:4100 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:35,749] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765, > HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2014-10-31 06:27:35,750] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : /172.31.7.214:9765 > {org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2014-10-31 06:28:30,604] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-61 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:32,606] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-65 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-73 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-69 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-64 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:33,609] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-75 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:34,610] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-60 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:34,611] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-62 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2014-10-31 06:28:34,611] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-67 > {org.apache.synapse.transport.passthru.SourceHandler} > > Need to restart the ELB to recover from this. Any idea what's going on? > Is this a known issue? Can provide the full log if needed. > > Thanks. > /Gayashan > > -- > *Gayashan Amarasinghe* > Software Engineer | Platform TG > WSO2, Inc. | http://wso2.com > lean. enterprise. middleware > > Mobile : +94718314517 > Blog : gayashan-a.blogspot.com > -- *Kishanthan Thangarajah* Senior Software Engineer, Platform Technologies Team, WSO2, Inc. lean.enterprise.middleware Mobile - +94773426635 Blog - *http://kishanthan.wordpress.com <http://kishanthan.wordpress.com>* Twitter - *http://twitter.com/kishanthan <http://twitter.com/kishanthan>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
