I think we have manged to find the root cause for the closedChannelException in ELB. This is due to the specific handling of GET request for favicon.ico that comes from the browser and the handing of this request leads to mess of the state machine of the transport. So, we have change the logic such that to handle that sort of request as we do with wsdls requests in PassThroughNHttpGetProcessor. We are testing the fix at the moment.
On Thu, Aug 22, 2013 at 9:35 AM, Jeewantha Dharmaparakrama < [email protected]> wrote: > > > > On Wed, Aug 21, 2013 at 7:12 PM, Evanthika Amarasiri > <[email protected]>wrote: > >> Hi Jeewantha, >> >> I have enabled wire logs in ELB & sent you setup information on a >> separate email. >> > Thanks a lot. > >> >> Regards, >> Evanthika >> >> >> On Wed, Aug 21, 2013 at 4:07 PM, Jeewantha Dharmaparakrama < >> [email protected]> wrote: >> >>> >>> >>> >>> On Mon, Aug 19, 2013 at 2:41 PM, Jeewantha Dharmaparakrama < >>> [email protected]> wrote: >>> >>>> Hi Evanthika, >>>> >>>> Can you kindly enable wire logs in the ELB in EC2 cluster and provide >>>> the logs when this exception is thrown? >>>> >>> The error logs are kinda urgent. It would be easy if we get access to >>> the QA EC2 setup. >>> Jeewantha. >>> >>>> >>>> Thanks, >>>> Jeewantha. >>>> >>>> >>>> On Fri, Aug 16, 2013 at 4:22 PM, Afkham Azeez <[email protected]> wrote: >>>> >>>>> It was a configuration issue where the subDomain wasn't properly >>>>> getting set. >>>>> >>>>> >>>>> On Fri, Aug 16, 2013 at 1:07 PM, Jeewantha Dharmaparakrama < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Folks, >>>>>> >>>>>> LB seems to be unstable when clustered with latest 4.2.0 kernal packs >>>>>> [1]. Though this was first thought of as a PTT issue (see thread@dev- >>>>>> "java.nio.channels.ClosedChannelException >>>>>> thrown time to time on ELB console") there seems to be some >>>>>> inconstancy when nodes are joined to the cluster which we are currently >>>>>> looking into. >>>>>> >>>>>> When a management node is added to the cluster, only the following >>>>>> log message is printed in sout. >>>>>> >>>>>> INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - >>>>>> Added member: Host:127.0.0.1, Remote Host:null, Port: 3999, HTTP:-1, >>>>>> HTTPS:-1, Domain: null, Sub-domain:null, Active:true >>>>>> >>>>>> This was checked with the packs built on 5th and 9th august [2] and >>>>>> as per discussion with Azeez and Sameera, the products do not get >>>>>> clustered >>>>>> properly. When management console is accessed, a SOAP Fault pops up on >>>>>> the >>>>>> browser as >>>>>> >>>>>> <soapenv:Envelope><soapenv: >>>>>> Body><nullResponse><Timestamp>Thu Aug 15 2013 16:55:05 GMT+0530 >>>>>> (IST)</Timestamp><Ack>Failure</Ack><Errors><ShortMessage>Gateway >>>>>> Error</ShortMessage><LongMessage>No application members available >>>>>> </LongMessage><ErrorCode>500 >>>>>> </ErrorCode><SeverityCode>Error</SeverityCode><ErrorClassification>RequestError</ErrorClassification></Errors><ServiceName>null</ServiceName><ResponseCode>null</ResponseCode><ContentType>null</ContentType><Version>1.5.1</Version></nullResponse></soapenv:Body></soapenv:Envelope> >>>>>> >>>>>> and with an Exception at the ELB backend >>>>>> >>>>>> [2013-08-15 16:55:05,407] ERROR - TenantAwareLoadBalanceEndpoint No >>>>>> application members available >>>>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Illegal incoming >>>>>> connection state: REQUEST_DONE . Possibly two send backs are happening >>>>>> for >>>>>> the same request >>>>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Trying to write >>>>>> response body while the handler is in an inconsistent state REQUEST_DONE >>>>>> [2013-08-15 16:55:05,910] ERROR - TenantAwareLoadBalanceEndpoint No >>>>>> application members available >>>>>> [2013-08-15 16:57:28,353] ERROR - TenantAwareLoadBalanceEndpoint No >>>>>> application members available >>>>>> [2013-08-15 16:57:28,708] WARN - SourceHandler Illegal incoming >>>>>> connection state: REQUEST_DONE . Possibly two send backs are happening >>>>>> for >>>>>> the same request >>>>>> [2013-08-15 16:57:28,709] WARN - SourceHandler Trying to write >>>>>> response body while the handler is in an inconsistent state REQUEST_DONE >>>>>> [2013-08-15 16:57:28,709] ERROR - TenantAwareLoadBalanceEndpoint No >>>>>> application members available >>>>>> [2013-08-15 16:57:28,710] ERROR - SourceHandler Unexpected I/O error: >>>>>> java.nio.channels.ClosedChannelException >>>>>> java.nio.channels.ClosedChannelException >>>>>> at >>>>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:135) >>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:326) >>>>>> at >>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331) >>>>>> at >>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) >>>>>> at >>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> ELB tries to send a request to the management node but the >>>>>> application member is not found, hence gets a ClosedChannelException. >>>>>> >>>>>> [1] https://wso2.org/jira/browse/LB-74 >>>>>> [2] >>>>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/05-08-2013/<http://www.google.com/url?q=http%3A%2F%2Fec2-54-224-94-128.compute-1.amazonaws.com%2F05-08-2013%2F&sa=D&sntz=1&usg=AFQjCNGFLcc1dySI1P2yUGSGveqEpgXIug> >>>>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/09-08-2013/ >>>>>> >>>>>> Jeewantha. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeewantha Dharmaparakrama >>>>>> Software Engineer; WSO2, Inc.; http://wso2.com/ >>>>>> Phone : (+94) 774726790 >>>>>> Skype : prasad.jeewantha >>>>>> LinkedIn : http://www.linkedin.com/in/jeewanthad >>>>>> Twitter: https://twitter.com/jeewamp >>>>>> Blog: http://jeewanthad.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Afkham Azeez* >>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>> * <http://www.apache.org/>** >>>>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>>>> twitter: >>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>> * >>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>>>> * >>>>> * >>>>> *Lean . Enterprise . Middleware* >>>>> >>>> >>>> >>>> >>>> -- >>>> Jeewantha Dharmaparakrama >>>> Software Engineer; WSO2, Inc.; http://wso2.com/ >>>> Phone : (+94) 774726790 >>>> Skype : prasad.jeewantha >>>> LinkedIn : http://www.linkedin.com/in/jeewanthad >>>> Twitter: https://twitter.com/jeewamp >>>> Blog: http://jeewanthad.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> Jeewantha Dharmaparakrama >>> Software Engineer; WSO2, Inc.; http://wso2.com/ >>> Phone : (+94) 774726790 >>> Skype : prasad.jeewantha >>> LinkedIn : http://www.linkedin.com/in/jeewanthad >>> Twitter: https://twitter.com/jeewamp >>> Blog: http://jeewanthad.blogspot.com/ >>> >> >> > > > -- > Jeewantha Dharmaparakrama > Software Engineer; WSO2, Inc.; http://wso2.com/ > Phone : (+94) 774726790 > Skype : prasad.jeewantha > LinkedIn : http://www.linkedin.com/in/jeewanthad > Twitter: https://twitter.com/jeewamp > Blog: http://jeewanthad.blogspot.com/ > -- Kasun Indrasiri Software Architect WSO2, Inc.; http://wso2.com lean.enterprise.middleware cell: +94 71 536 4128 Blog : http://kasunpanorama.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
