Hi all,
For Carbon testing we have a worker-mgt cluster fronted by ELB and requests
keep coming in from a jmeter client. During this if one (or more) of the
worker nodes were shutdown, after some time the ELB stops sending requests
to the nodes and the connection times out. Following log gets printed in
the ELB.
TID: [0] [ELB] [2014-10-31 06:27:32,517] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port:
4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
Active:true . Error Code: 101503
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:32,519] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Dropping the faulty/unreachable Member with Domain:wso2.as.domain,
Host:172.31.7.214, Port:4100
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:32,738] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:172.31.0.128, Remote Host:null, Port: 4100, HTTP:9763,
HTTPS:9443, Domain: wso2.as.domain, Sub-domain:worker, Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:32,740] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for : /172.31.7.214:9765
{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2014-10-31 06:27:32,743] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port:
4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
Active:true . Error Code: 101503
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:32,745] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Dropping the faulty/unreachable Member with Domain:wso2.as.domain,
Host:172.31.7.214, Port:4100
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:33,518] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765,
HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:33,520] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for : /172.31.7.214:9765
{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2014-10-31 06:27:33,523] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for : /172.31.7.214:9765
{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2014-10-31 06:27:33,744] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765,
HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:33,745] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port:
4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
Active:true . Error Code: 101503
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:33,745] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for : /172.31.7.214:9765
{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2014-10-31 06:27:33,747] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Dropping the faulty/unreachable Member with Domain:wso2.as.domain,
Host:172.31.7.214, Port:4100
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:34,746] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:172.31.0.128, Remote Host:null, Port: 4100, HTTP:9763,
HTTPS:9443, Domain: wso2.as.domain, Sub-domain:worker, Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:34,748] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed to send message to Member Host:172.31.7.214, Remote Host:null, Port:
4100, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
Active:true . Error Code: 101503
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:34,750] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Dropping the faulty/unreachable Member with Domain:wso2.as.domain,
Host:172.31.7.214, Port:4100
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:35,749] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:172.31.7.214, Remote Host:null, Port: 4100, HTTP:9765,
HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker, Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2014-10-31 06:27:35,750] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for : /172.31.7.214:9765
{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2014-10-31 06:28:30,604] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-61
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:32,606] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-65
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-73
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-69
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:33,608] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-64
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:33,609] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-75
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:34,610] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-60
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:34,611] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-62
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2014-10-31 06:28:34,611] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-67
{org.apache.synapse.transport.passthru.SourceHandler}
Need to restart the ELB to recover from this. Any idea what's going on?
Is this a known issue? Can provide the full log if needed.
Thanks.
/Gayashan
--
*Gayashan Amarasinghe*
Software Engineer | Platform TG
WSO2, Inc. | http://wso2.com
lean. enterprise. middleware
Mobile : +94718314517
Blog : gayashan-a.blogspot.com
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
