Hi again, On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka <[email protected]> wrote:
> Hi Oleg, > > I can reproduce the problem with the attached JUnit test case. This test case > doesn't use any Axis2/Synapse related code. It only requires HTTP Core and > HTTP client. I used HTTP Core 4.2.4 for testing. > > The test case starts a ListeningIOReactor, does a simple HTTP > request-response exchange and then terminates the reactor. Then after a > couple of seconds, it tries to start the IO reactor again and the exception > occurs. Here's the full output: > > Round 1 > Listener started on port: 8280 > Connected > Request received > Response submitted > Waiting - Shutdown in 5 secs > Shutdown > > … wait 2 seconds... > > Round 2 > Listener started on port: 8280 > Waiting - Shutdown in 5 secs > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) > at > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238) > at > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374) > at > org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101) > Shutdown I've also noticed that the problem doesn't occur if we do not send any messages in the 1st round. I verified this behavior with Synapse too. If a test case doesn't send any messages to Synapse, the test case that follows will run without any issues. Thanks, Hiranya > > Thanks, > Hiranya > > <HttpCoreTest.java> > > On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <[email protected]> wrote: > >> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote: >>> It looks like the socket goes into the CLOSE_WAIT state after shutting >>> down the IO reactor and remains there until the JVM it self is killed. >>> Since all automated test cases are executed within a single JVM >>> instance, this issue continues to occur. Looks like something is not >>> getting cleaned up properly with the latest HTTP Core. >>> >>> >>> Thanks, >>> Hiranya >>> >> >> Hiranya >> >> I need a more specific evidence of HttpCore doing something wrong. Could >> you please reproduce the issue with a test app, preferably not dependent >> on Synapse? >> >> Oleg >> >>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka >>> <[email protected]> wrote: >>> >>>> Hi Oleg, >>>> >>>> >>>> I've done the necessary code changes locally and it seems to work >>>> fine. The HTTPS sender now works as expected. But when running >>>> automated tests, I frequently get the following exception: >>>> >>>> >>>> 34 [HttpCoreNIOListener] WARN >>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may >>>> be unstable: IOReactor encountered a checked exception : Address >>>> already in use >>>> java.net.BindException: Address already in use >>>> at sun.nio.ch.Net.bind(Native Method) >>>> at >>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124) >>>> at >>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) >>>> at >>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52) >>>> at >>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238) >>>> at >>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144) >>>> at >>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374) >>>> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener >>>> $2.run(HttpCoreNIOListener.java:254) >>>> at java.lang.Thread.run(Thread.java:680) >>>> >>>> >>>> This is seen in both NHTTP unit tests and Synapse integration tests. >>>> Any idea what is going on here? Synapse cleans up the IO reactor by >>>> calling ioReactor.shutdown(). Is there anything else we need to do >>>> to clean up the reactor properly? >>>> >>>> >>>> Thanks, >>>> Hiranya >>>> >>>> >>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka >>>> <[email protected]> wrote: >>>> >>>>> HI Oleg, >>>>> >>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <[email protected]> >>>>> wrote: >>>>> >>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote: >>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri >>>>>>> <[email protected]> wrote: >>>>>>> >>>>>>>> May be we should enable the SSl debug logs and see what's >>>>>>>> going on ? >>>>>>> >>>>>>> I ran some more tests and it looks like the HTTPS sender >>>>>>> doesn't work with HTTP Core 4.2. It just sends out the >>>>>>> messages without any SSL security. I can even monitor the >>>>>>> message in plain text using TCPMon. Looks like some SSL >>>>>>> related code in the Synapse HTTPS sender doesn't work properly >>>>>>> with the new HTTP Core. >>>>>>> >>>>>>> Thanks, >>>>>>> Hiranya >>>>>>> >>>>>> >>>>>> Hiranya >>>>>> >>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs? >>>>> >>>>> >>>>> No not yet. I just tried changing the HTTP Core dependency version >>>>> to begin with. I guess migrating off the deprecated APIs is the >>>>> next logical step. >>>>> >>>>> >>>>> Thanks, >>>>> Hiranya >>>>> >>>>>> >>>>>> Oleg >>>>>> >>>>>> PS: By the way, you should probably consider migrating straight >>>>>> to 4.3 >>>>>> at this point. >>>>>> >>>>>> >>>>>>>> >>>>>>>> Rajika >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka >>>>>>>> <[email protected]> wrote: >>>>>>>> Hi Folks, >>>>>>>> >>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried >>>>>>>> using 4.2 instead (just changed the version of the maven >>>>>>>> dependency). Almost everything worked fine, except for a >>>>>>>> couple of integration test failures. In both these tests >>>>>>>> Synapse uses HTTPS to contact the backend services, so I >>>>>>>> suppose that's where things went wrong. The backend server >>>>>>>> threw the following exception in the process: >>>>>>>> >>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O >>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized >>>>>>>> SSL message, plaintext connection? >>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message, >>>>>>>> plaintext connection? >>>>>>>> at >>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152) >>>>>>>> at >>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806) >>>>>>>> at >>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721) >>>>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607) >>>>>>>> at >>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228) >>>>>>>> at >>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263) >>>>>>>> at >>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) >>>>>>>> at >>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) >>>>>>>> at java.lang.Thread.run(Thread.java:680) >>>>>>>> >>>>>>>> I also tried upgrading to the latest HTTP Core version >>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most >>>>>>>> of the integration tests failed too. It turns out only the >>>>>>>> very first integration test ran successfully. Everything >>>>>>>> that followed threw a bind exception. Looks like something >>>>>>>> isn't getting cleaned up properly (although the logs >>>>>>>> indicate that Synapse transport listeners are shutting down >>>>>>>> cleanly after each test case). >>>>>>>> >>>>>>>> Does anybody got an idea what's going on? Do we need to do >>>>>>>> any code changes to migrate to the latest HTTP Core? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Hiranya >>>>>>>> -- >>>>>>>> Hiranya Jayathilaka >>>>>>>> Mayhem Lab/RACE Lab; >>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>>>>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>>>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Hiranya Jayathilaka >>>>>>> Mayhem Lab/RACE Lab; >>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>>>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>> >>>>> -- >>>>> Hiranya Jayathilaka >>>>> Mayhem Lab/RACE Lab; >>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>> >>>>> >>>> >>>> -- >>>> Hiranya Jayathilaka >>>> Mayhem Lab/RACE Lab; >>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>>> Blog: http://techfeast-hiranya.blogspot.com >>>> >>>> >>> >>> -- >>> Hiranya Jayathilaka >>> Mayhem Lab/RACE Lab; >>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu >>> E-mail: [email protected]; Mobile: +1 (805) 895-7443 >>> Blog: http://techfeast-hiranya.blogspot.com >>> >>> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > -- > Hiranya Jayathilaka > Mayhem Lab/RACE Lab; > Dept. of Computer Science, UCSB; http://cs.ucsb.edu > E-mail: [email protected]; Mobile: +1 (805) 895-7443 > Blog: http://techfeast-hiranya.blogspot.com > -- Hiranya Jayathilaka Mayhem Lab/RACE Lab; Dept. of Computer Science, UCSB; http://cs.ucsb.edu E-mail: [email protected]; Mobile: +1 (805) 895-7443 Blog: http://techfeast-hiranya.blogspot.com
