Hi Oleg, I tested the patch. When I haven't explicitly set the connection timeout, I got the same exception, java.net.ConnectException: Connection timed out, at concurrency = 3000, messages/thread = 100 intermittently. One time I got it in less than 17 seconds. This is contrary to the Javadoc of Socket#connect() method [1] which says "*A timeout of zero is interpreted as an infinite timeout*".
Then when I explicitly set it to 5 seconds as follows I was unable to reproduce the exception although I tried several times. socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5); The other doubt I have is; although I'm getting a java.net.ConnectException, according to [1], this method should throw SocketTimeoutException in case of a timeout. Further, JavaDoc of ConnectException (the one I'm actually getting) says it signals that *"an error occurred while attempting to connect a socket to a remote address and port. Typically, the connection was refused remotely (e.g., no process is listening on the remote address/port)." * Therefore, I believe that the actual cause of this error is server's ListeningIOReactor not being able to react to incoming connection requests. I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor. Maybe increasing it further would solve this issue. I will test this further and let you know. [1] http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29 [2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html Thanks, Sadeep On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <[email protected]>wrote: > Hi Oleg, > > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <[email protected]>wrote: > >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote: >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote: >> > > Hi, >> > > >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <[email protected]> >> wrote: >> > > >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote: >> > > > > Hi, >> > > > > >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski < >> [email protected]> >> > > > wrote: >> > > > > >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote: >> > > > > > > Hi, >> > > > > > > >> > > > > > > I have written an HTTP server using HTTPCore-NIO and currently >> > > > testing it >> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS values >> so >> > > > far. But >> > > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency >> > > > level, 50 >> > > > > > > messages per thread, I'm getting the following exception >> several >> > > > times >> > > > > > from >> > > > > > > httpcore-ab and it results in "Failed requests" in the >> httpcore-ab >> > > > > > output. >> > > > > > > No exceptions are displayed from the server side. Hence, I >> think the >> > > > > > server >> > > > > > > is rejecting some socket connections because it's already >> fully >> > > > loaded. >> > > > > > > >> > > > > > > I'm running the server and load generator on two server >> machines >> > > > that are >> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU >> cores, 16 GB >> > > > > > memory >> > > > > > > etc. >> > > > > > > >> > > > > > > I have increased the socket timeout in both httpcore-ab and >> server >> > > > side >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the >> > > > reactor. >> > > > > > > Messages received by the server are processed in a separate >> worker >> > > > pool >> > > > > > > which has ~500 threads. I have also increased the open file >> limit of >> > > > my >> > > > > > OS >> > > > > > > (Debian) to 65535. >> > > > > > > >> > > > > > > Is there any tuning parameter of HTTPCore that I can used to >> get rid >> > > > of >> > > > > > > this exception? Perhaps, something similar to >> "acceptQueueSize" in >> > > > Jetty >> > > > > > > [2]? >> > > > > > > >> > > > > > >> > > > > > Hi Sadeep >> > > > > > >> > > > > > A couple of points in no particular logical order. >> > > > > > >> > > > > > (1) HttpCore does not depend on a logging toolkit and does not >> do any >> > > > > > logging per default. If it ever encounters an abnormal >> situation, >> > > > > > instead of logging such an event it throws an exception and >> terminates. >> > > > > > Quite often this is not desirable, so one can provide an >> exception >> > > > > > handler that can log using a a logging toolkit of choice or >> handle a >> > > > > > particular type of exception in an application specific way. >> > > > > > >> > > > > > If the server did not terminate under with an exception, I kind >> of >> > > > > > suspect it was still trying to chew through the load without >> > > > > > encountering anything abnormal. Unless you have added a custom >> > > > exception >> > > > > > handler. >> > > > > > >> > > > > > (2) HttpCore does not have a concept of accepted connection >> queue. As >> > > > > > soon as the listener accepts an incoming connection its channel >> gets >> > > > > > immediately added to the underlying i/o selector. No >> intermediate >> > > > > > queuing takes place. >> > > > > >> > > > > >> > > > > > (3) Given that the connection times out on the client, this >> sounds more >> > > > > > like a client side issue. Was connection really suck in >> > > > > > Socket#socketConnect for 360000? >> > > > > > >> > > > > >> > > > > I'm using httpcore-ab [1] as the client. I have set socket >> timeout of >> > > > using >> > > > > -t 360000, and changed the connection timeout with the patch >> shown below. >> > > > > But I'm starting to get this error well before 360000 millis past >> the >> > > > test >> > > > > startup. Therefore, I there is no possibility that the connection >> attempt >> > > > > hangs for 360000 millis before printing this exception. I was >> thinking >> > > > > whether there are upper bounds for these limits that cause my >> settings to >> > > > > be neglected. >> > > > > >> > > > >> > > > You see. If java.net.ConnectException: Connection timed out is >> thrown by >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse, >> > > > there must be something wrong with the client side. >> > > > >> > > >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that >> it >> > > doesn't wait 360000 millis before throwing this exception. >> > > >> > >> >> Hi Sadeep >> >> It did turn out to be an issue with the way HttpCore AB initialized >> connections. It was basically using an old (Java 1.0) socket constructor >> whose behavior can differ between OS platforms and possibly between JRE >> major releases with regards to connect timeout handling. The connect >> timeout value set in the HttpParams simply had no effect and therefore >> connections in your tests were timing out before 360000 milliseconds >> elapsed (probably using some sort of a system default). >> >> I committed a fix for the problem >> >> http://svn.apache.org/viewvc?rev=1238566&view=rev >> >> Please review >> > > Many thanks for looking into this. I will review this at my earliest and > let you know the outcome. > > Thanks, > Sadeep > > >> >> Oleg >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > > -- > > Sadeep Jayasumana**** > > Email: [email protected]**** > > Phone: +94-77-2266507 > > > > -- Sadeep Jayasumana**** Email: [email protected]**** Phone: +94-77-2266507
