On Wed, Aug 21, 2013 at 7:39 AM, Sameera Jayasoma <[email protected]> wrote:
> Can we have an update on the ELB issues? > This issue has been dragging for some time and its not been easy to fix it cause the error is thrown from httpcore-nio runtime. While an HTTP request being served, the connection gets closed abruptly and the server continues to use a closed connection, hence the error is thrown. The problem is even worse since its intermittent and cant get reproduced locally. This can either be a bug in httpcore-nio or not be a bug at all. When a connection is broken due to network inconsistencies, it is reasonable for the server to throw an Exception. But at the same time, httocore can erroneously be using a closed connection. We cant analyze the run time behaviour either coz the issue does not get reproduced locally. Using EC2 setup used by QA team for debugging purposes is not practical. So its a good idea if we can have a clone of the Ec2 set-up for debugging purposes. Jeewantha > > On Mon, Aug 19, 2013 at 2:41 PM, Jeewantha Dharmaparakrama < > [email protected]> wrote: > >> Hi Evanthika, >> >> Can you kindly enable wire logs in the ELB in EC2 cluster and provide the >> logs when this exception is thrown? >> >> Thanks, >> Jeewantha. >> >> >> On Fri, Aug 16, 2013 at 4:22 PM, Afkham Azeez <[email protected]> wrote: >> >>> It was a configuration issue where the subDomain wasn't properly getting >>> set. >>> >>> >>> On Fri, Aug 16, 2013 at 1:07 PM, Jeewantha Dharmaparakrama < >>> [email protected]> wrote: >>> >>>> Hi Folks, >>>> >>>> LB seems to be unstable when clustered with latest 4.2.0 kernal packs >>>> [1]. Though this was first thought of as a PTT issue (see thread@dev - >>>> "java.nio.channels.ClosedChannelException thrown time to time on ELB >>>> console") there seems to be some inconstancy when nodes are joined to >>>> the cluster which we are currently looking into. >>>> >>>> When a management node is added to the cluster, only the following log >>>> message is printed in sout. >>>> >>>> INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - >>>> Added member: Host:127.0.0.1, Remote Host:null, Port: 3999, HTTP:-1, >>>> HTTPS:-1, Domain: null, Sub-domain:null, Active:true >>>> >>>> This was checked with the packs built on 5th and 9th august [2] and as >>>> per discussion with Azeez and Sameera, the products do not get clustered >>>> properly. When management console is accessed, a SOAP Fault pops up on the >>>> browser as >>>> >>>> <soapenv:Envelope><soapenv: >>>> Body><nullResponse><Timestamp>Thu Aug 15 2013 16:55:05 GMT+0530 >>>> (IST)</Timestamp><Ack>Failure</Ack><Errors><ShortMessage>Gateway >>>> Error</ShortMessage><LongMessage>No application members available >>>> </LongMessage><ErrorCode>500 >>>> </ErrorCode><SeverityCode>Error</SeverityCode><ErrorClassification>RequestError</ErrorClassification></Errors><ServiceName>null</ServiceName><ResponseCode>null</ResponseCode><ContentType>null</ContentType><Version>1.5.1</Version></nullResponse></soapenv:Body></soapenv:Envelope> >>>> >>>> and with an Exception at the ELB backend >>>> >>>> [2013-08-15 16:55:05,407] ERROR - TenantAwareLoadBalanceEndpoint No >>>> application members available >>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Illegal incoming >>>> connection state: REQUEST_DONE . Possibly two send backs are happening for >>>> the same request >>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Trying to write >>>> response body while the handler is in an inconsistent state REQUEST_DONE >>>> [2013-08-15 16:55:05,910] ERROR - TenantAwareLoadBalanceEndpoint No >>>> application members available >>>> [2013-08-15 16:57:28,353] ERROR - TenantAwareLoadBalanceEndpoint No >>>> application members available >>>> [2013-08-15 16:57:28,708] WARN - SourceHandler Illegal incoming >>>> connection state: REQUEST_DONE . Possibly two send backs are happening for >>>> the same request >>>> [2013-08-15 16:57:28,709] WARN - SourceHandler Trying to write >>>> response body while the handler is in an inconsistent state REQUEST_DONE >>>> [2013-08-15 16:57:28,709] ERROR - TenantAwareLoadBalanceEndpoint No >>>> application members available >>>> [2013-08-15 16:57:28,710] ERROR - SourceHandler Unexpected I/O error: >>>> java.nio.channels.ClosedChannelException >>>> java.nio.channels.ClosedChannelException >>>> at >>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:135) >>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:326) >>>> at >>>> org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331) >>>> at >>>> org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150) >>>> at >>>> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) >>>> at >>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) >>>> at java.lang.Thread.run(Thread.java:662) >>>> >>>> ELB tries to send a request to the management node but the application >>>> member is not found, hence gets a ClosedChannelException. >>>> >>>> [1] https://wso2.org/jira/browse/LB-74 >>>> [2] >>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/05-08-2013/<http://www.google.com/url?q=http%3A%2F%2Fec2-54-224-94-128.compute-1.amazonaws.com%2F05-08-2013%2F&sa=D&sntz=1&usg=AFQjCNGFLcc1dySI1P2yUGSGveqEpgXIug> >>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/09-08-2013/ >>>> >>>> Jeewantha. >>>> >>>> >>>> >>>> -- >>>> Jeewantha Dharmaparakrama >>>> Software Engineer; WSO2, Inc.; http://wso2.com/ >>>> Phone : (+94) 774726790 >>>> Skype : prasad.jeewantha >>>> LinkedIn : http://www.linkedin.com/in/jeewanthad >>>> Twitter: https://twitter.com/jeewamp >>>> Blog: http://jeewanthad.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> *Afkham Azeez* >>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> Member; Apache Software Foundation; http://www.apache.org/ >>> * <http://www.apache.org/>** >>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>> * >>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>> * >>> * >>> *Lean . Enterprise . Middleware* >>> >> >> >> >> -- >> Jeewantha Dharmaparakrama >> Software Engineer; WSO2, Inc.; http://wso2.com/ >> Phone : (+94) 774726790 >> Skype : prasad.jeewantha >> LinkedIn : http://www.linkedin.com/in/jeewanthad >> Twitter: https://twitter.com/jeewamp >> Blog: http://jeewanthad.blogspot.com/ >> > > > > -- > Sameera Jayasoma, > Architect, > > WSO2, Inc. (http://wso2.com) > email: [email protected] > blog: http://sameera.adahas.org > twitter: https://twitter.com/sameerajayasoma > flickr: http://www.flickr.com/photos/sameera-jayasoma/collections > > > Lean . Enterprise . Middleware > -- Jeewantha Dharmaparakrama Software Engineer; WSO2, Inc.; http://wso2.com/ Phone : (+94) 774726790 Skype : prasad.jeewantha LinkedIn : http://www.linkedin.com/in/jeewanthad Twitter: https://twitter.com/jeewamp Blog: http://jeewanthad.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
