On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote: > Hi Oleg, > > I tested the patch. When I haven't explicitly set the connection timeout, I > got the same exception, java.net.ConnectException: Connection timed out, at > concurrency = 3000, messages/thread = 100 intermittently. One time I got it > in less than 17 seconds. This is contrary to the Javadoc of > Socket#connect() method [1] which says "*A timeout of zero is interpreted > as an infinite timeout*". > > Then when I explicitly set it to 5 seconds as follows I was unable to > reproduce the exception although I tried several times. > > socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5); > > The other doubt I have is; although I'm getting a > java.net.ConnectException, according to [1], this method should throw > SocketTimeoutException in case of a timeout. Further, JavaDoc of > ConnectException (the one I'm actually getting) says it signals that *"an > error occurred while attempting to connect a socket to a remote address and > port. Typically, the connection was refused remotely (e.g., no process is > listening on the remote address/port)." * > > Therefore, I believe that the actual cause of this error is server's > ListeningIOReactor not being able to react to incoming connection requests. > I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor. > Maybe increasing it further would solve this issue. I will test this > further and let you know. >
ConnectException is a subclass of SocketTimeoutException and the queer thing the exception message clearly says "Connection timed out". So, I am still not entirely convinced this rules out a client side issue. Please also note that in its current implementation the DefaultListeningIOReactor utilizes only one thread and only one i/o selector for all incoming requests no matter how many worker threads are allocated for handling i/o on active channels. If you suspect this could be the issue, you may need to consider writing a custom ListeningIOReactor Oleg > [1] > http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29 > [2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html > > Thanks, > Sadeep > > On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana > <[email protected]>wrote: > > > Hi Oleg, > > > > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <[email protected]>wrote: > > > >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote: > >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote: > >> > > Hi, > >> > > > >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <[email protected]> > >> wrote: > >> > > > >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote: > >> > > > > Hi, > >> > > > > > >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski < > >> [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote: > >> > > > > > > Hi, > >> > > > > > > > >> > > > > > > I have written an HTTP server using HTTPCore-NIO and currently > >> > > > testing it > >> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS values > >> so > >> > > > far. But > >> > > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency > >> > > > level, 50 > >> > > > > > > messages per thread, I'm getting the following exception > >> several > >> > > > times > >> > > > > > from > >> > > > > > > httpcore-ab and it results in "Failed requests" in the > >> httpcore-ab > >> > > > > > output. > >> > > > > > > No exceptions are displayed from the server side. Hence, I > >> think the > >> > > > > > server > >> > > > > > > is rejecting some socket connections because it's already > >> fully > >> > > > loaded. > >> > > > > > > > >> > > > > > > I'm running the server and load generator on two server > >> machines > >> > > > that are > >> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU > >> cores, 16 GB > >> > > > > > memory > >> > > > > > > etc. > >> > > > > > > > >> > > > > > > I have increased the socket timeout in both httpcore-ab and > >> server > >> > > > side > >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the > >> > > > reactor. > >> > > > > > > Messages received by the server are processed in a separate > >> worker > >> > > > pool > >> > > > > > > which has ~500 threads. I have also increased the open file > >> limit of > >> > > > my > >> > > > > > OS > >> > > > > > > (Debian) to 65535. > >> > > > > > > > >> > > > > > > Is there any tuning parameter of HTTPCore that I can used to > >> get rid > >> > > > of > >> > > > > > > this exception? Perhaps, something similar to > >> "acceptQueueSize" in > >> > > > Jetty > >> > > > > > > [2]? > >> > > > > > > > >> > > > > > > >> > > > > > Hi Sadeep > >> > > > > > > >> > > > > > A couple of points in no particular logical order. > >> > > > > > > >> > > > > > (1) HttpCore does not depend on a logging toolkit and does not > >> do any > >> > > > > > logging per default. If it ever encounters an abnormal > >> situation, > >> > > > > > instead of logging such an event it throws an exception and > >> terminates. > >> > > > > > Quite often this is not desirable, so one can provide an > >> exception > >> > > > > > handler that can log using a a logging toolkit of choice or > >> handle a > >> > > > > > particular type of exception in an application specific way. > >> > > > > > > >> > > > > > If the server did not terminate under with an exception, I kind > >> of > >> > > > > > suspect it was still trying to chew through the load without > >> > > > > > encountering anything abnormal. Unless you have added a custom > >> > > > exception > >> > > > > > handler. > >> > > > > > > >> > > > > > (2) HttpCore does not have a concept of accepted connection > >> queue. As > >> > > > > > soon as the listener accepts an incoming connection its channel > >> gets > >> > > > > > immediately added to the underlying i/o selector. No > >> intermediate > >> > > > > > queuing takes place. > >> > > > > > >> > > > > > >> > > > > > (3) Given that the connection times out on the client, this > >> sounds more > >> > > > > > like a client side issue. Was connection really suck in > >> > > > > > Socket#socketConnect for 360000? > >> > > > > > > >> > > > > > >> > > > > I'm using httpcore-ab [1] as the client. I have set socket > >> timeout of > >> > > > using > >> > > > > -t 360000, and changed the connection timeout with the patch > >> shown below. > >> > > > > But I'm starting to get this error well before 360000 millis past > >> the > >> > > > test > >> > > > > startup. Therefore, I there is no possibility that the connection > >> attempt > >> > > > > hangs for 360000 millis before printing this exception. I was > >> thinking > >> > > > > whether there are upper bounds for these limits that cause my > >> settings to > >> > > > > be neglected. > >> > > > > > >> > > > > >> > > > You see. If java.net.ConnectException: Connection timed out is > >> thrown by > >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse, > >> > > > there must be something wrong with the client side. > >> > > > > >> > > > >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that > >> it > >> > > doesn't wait 360000 millis before throwing this exception. > >> > > > >> > > >> > >> Hi Sadeep > >> > >> It did turn out to be an issue with the way HttpCore AB initialized > >> connections. It was basically using an old (Java 1.0) socket constructor > >> whose behavior can differ between OS platforms and possibly between JRE > >> major releases with regards to connect timeout handling. The connect > >> timeout value set in the HttpParams simply had no effect and therefore > >> connections in your tests were timing out before 360000 milliseconds > >> elapsed (probably using some sort of a system default). > >> > >> I committed a fix for the problem > >> > >> http://svn.apache.org/viewvc?rev=1238566&view=rev > >> > >> Please review > >> > > > > Many thanks for looking into this. I will review this at my earliest and > > let you know the outcome. > > > > Thanks, > > Sadeep > > > > > >> > >> Oleg > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > > > > -- > > > > Sadeep Jayasumana**** > > > > Email: [email protected]**** > > > > Phone: +94-77-2266507 > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
