It looks like the socket goes into the CLOSE_WAIT state after shutting down the IO reactor and remains there until the JVM it self is killed. Since all automated test cases are executed within a single JVM instance, this issue continues to occur. Looks like something is not getting cleaned up properly with the latest HTTP Core.
Thanks, Hiranya On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka <[email protected]> wrote: > Hi Oleg, > > I've done the necessary code changes locally and it seems to work fine. The > HTTPS sender now works as expected. But when running automated tests, I > frequently get the following exception: > > 34 [HttpCoreNIOListener] WARN > org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may be > unstable: IOReactor encountered a checked exception : Address already in use > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) > at > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238) > at > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374) > at > org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:254) > at java.lang.Thread.run(Thread.java:680) > > This is seen in both NHTTP unit tests and Synapse integration tests. Any idea > what is going on here? Synapse cleans up the IO reactor by calling > ioReactor.shutdown(). Is there anything else we need to do to clean up the > reactor properly? > > Thanks, > Hiranya > > On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka <[email protected]> > wrote: > >> HI Oleg, >> >> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <[email protected]> wrote: >> >>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote: >>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri >>>> <[email protected]> wrote: >>>> >>>>> May be we should enable the SSl debug logs and see what's going on ? >>>> >>>> I ran some more tests and it looks like the HTTPS sender doesn't work with >>>> HTTP Core 4.2. It just sends out the messages without any SSL security. I >>>> can even monitor the message in plain text using TCPMon. Looks like some >>>> SSL related code in the Synapse HTTPS sender doesn't work properly with >>>> the new HTTP Core. >>>> >>>> Thanks, >>>> Hiranya >>>> >>> >>> Hiranya >>> >>> Have you migrated off the deprecated APIs to new 4.2 APIs? >> >> No not yet. I just tried changing the HTTP Core dependency version to begin >> with. I guess migrating off the deprecated APIs is the next logical step. >> >> Thanks, >> Hiranya >> >>> >>> Oleg >>> >>> PS: By the way, you should probably consider migrating straight to 4.3 >>> at this point. >>> >>> >>>>> >>>>> Rajika >>>>> >>>>> >>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka >>>>> <[email protected]> wrote: >>>>> Hi Folks, >>>>> >>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 >>>>> instead (just changed the version of the maven dependency). Almost >>>>> everything worked fine, except for a couple of integration test failures. >>>>> In both these tests Synapse uses HTTPS to contact the backend services, >>>>> so I suppose that's where things went wrong. The backend server threw the >>>>> following exception in the process: >>>>> >>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR >>>>> ServerHandler I/O error: Unrecognized SSL message, plaintext connection? >>>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext >>>>> connection? >>>>> at >>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152) >>>>> at >>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806) >>>>> at >>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721) >>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263) >>>>> at >>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370) >>>>> at >>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229) >>>>> at >>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) >>>>> at >>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) >>>>> at >>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) >>>>> at java.lang.Thread.run(Thread.java:680) >>>>> >>>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this >>>>> version NHTTP unit tests failed and most of the integration tests failed >>>>> too. It turns out only the very first integration test ran successfully. >>>>> Everything that followed threw a bind exception. Looks like something >>>>> isn't getting cleaned up properly (although the logs indicate that >>>>> Synapse transport listeners are shutting down cleanly after each test >>>>> case). >>>>> >>>>> Does anybody got an idea what's going on? Do we need to do any code >>>>> changes to migrate to the latest HTTP Core? >>>>> >>>>> Thanks, >>>>> Hiranya >>>>> -- >>>>> Hiranya Jayathilaka >>>>> Mayhem Lab/RACE Lab; >>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>> >>>>> >>>> >>>> -- >>>> Hiranya Jayathilaka >>>> Mayhem Lab/RACE Lab; >>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>> Blog: http://techfeast-hiranya.blogspot.com >>>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> -- >> Hiranya Jayathilaka >> Mayhem Lab/RACE Lab; >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >> Blog: http://techfeast-hiranya.blogspot.com >> > > -- > Hiranya Jayathilaka > Mayhem Lab/RACE Lab; > Dept. of Computer Science, UCSB; http://cs.ucsb.edu > E-mail: [email protected]; Mobile: +1 (805) 895-7443 > Blog: http://techfeast-hiranya.blogspot.com > -- Hiranya Jayathilaka Mayhem Lab/RACE Lab; Dept. of Computer Science, UCSB; http://cs.ucsb.edu E-mail: [email protected]; Mobile: +1 (805) 895-7443 Blog: http://techfeast-hiranya.blogspot.com
