Hi Oleg, On Wed, Feb 1, 2012 at 5:38 PM, Oleg Kalnichevski <[email protected]> wrote:
> On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote: > > Hi Oleg, > > > > I tested the patch. When I haven't explicitly set the connection > timeout, I > > got the same exception, java.net.ConnectException: Connection timed out, > at > > concurrency = 3000, messages/thread = 100 intermittently. One time I got > it > > in less than 17 seconds. This is contrary to the Javadoc of > > Socket#connect() method [1] which says "*A timeout of zero is interpreted > > as an infinite timeout*". > > > > Then when I explicitly set it to 5 seconds as follows I was unable to > > reproduce the exception although I tried several times. > > > > socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5); > > > > The other doubt I have is; although I'm getting a > > java.net.ConnectException, according to [1], this method should throw > > SocketTimeoutException in case of a timeout. Further, JavaDoc of > > ConnectException (the one I'm actually getting) says it signals that *"an > > error occurred while attempting to connect a socket to a remote address > and > > port. Typically, the connection was refused remotely (e.g., no process is > > listening on the remote address/port)." * > > > > Therefore, I believe that the actual cause of this error is server's > > ListeningIOReactor not being able to react to incoming connection > requests. > > I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor. > > Maybe increasing it further would solve this issue. I will test this > > further and let you know. > > > > ConnectException is a subclass of SocketTimeoutException and the queer > thing the exception message clearly says "Connection timed out". It turns out that ConnectException is not a subclass of SocketTimeoutException. That's why I'm confused. So, I > am still not entirely convinced this rules out a client side issue. > > Please also note that in its current implementation the > DefaultListeningIOReactor utilizes only one thread and only one i/o > selector for all incoming requests no matter how many worker threads are > allocated for handling i/o on active channels. If you suspect this could > be the issue, you may need to consider writing a custom > ListeningIOReactor > Got the idea. Thanks for the explanation. Thanks, Sadeep > > Oleg > > > [1] > > > http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29 > > [2] > http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html > > > > Thanks, > > Sadeep > > > > On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana < > [email protected]>wrote: > > > > > Hi Oleg, > > > > > > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <[email protected] > >wrote: > > > > > >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote: > > >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote: > > >> > > Hi, > > >> > > > > >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski < > [email protected]> > > >> wrote: > > >> > > > > >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote: > > >> > > > > Hi, > > >> > > > > > > >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski < > > >> [email protected]> > > >> > > > wrote: > > >> > > > > > > >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote: > > >> > > > > > > Hi, > > >> > > > > > > > > >> > > > > > > I have written an HTTP server using HTTPCore-NIO and > currently > > >> > > > testing it > > >> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS > values > > >> so > > >> > > > far. But > > >> > > > > > > when I try to send 5KB HTTP messages with 2500 and > concurrency > > >> > > > level, 50 > > >> > > > > > > messages per thread, I'm getting the following exception > > >> several > > >> > > > times > > >> > > > > > from > > >> > > > > > > httpcore-ab and it results in "Failed requests" in the > > >> httpcore-ab > > >> > > > > > output. > > >> > > > > > > No exceptions are displayed from the server side. Hence, I > > >> think the > > >> > > > > > server > > >> > > > > > > is rejecting some socket connections because it's already > > >> fully > > >> > > > loaded. > > >> > > > > > > > > >> > > > > > > I'm running the server and load generator on two server > > >> machines > > >> > > > that are > > >> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU > > >> cores, 16 GB > > >> > > > > > memory > > >> > > > > > > etc. > > >> > > > > > > > > >> > > > > > > I have increased the socket timeout in both httpcore-ab > and > > >> server > > >> > > > side > > >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated > for the > > >> > > > reactor. > > >> > > > > > > Messages received by the server are processed in a > separate > > >> worker > > >> > > > pool > > >> > > > > > > which has ~500 threads. I have also increased the open > file > > >> limit of > > >> > > > my > > >> > > > > > OS > > >> > > > > > > (Debian) to 65535. > > >> > > > > > > > > >> > > > > > > Is there any tuning parameter of HTTPCore that I can used > to > > >> get rid > > >> > > > of > > >> > > > > > > this exception? Perhaps, something similar to > > >> "acceptQueueSize" in > > >> > > > Jetty > > >> > > > > > > [2]? > > >> > > > > > > > > >> > > > > > > > >> > > > > > Hi Sadeep > > >> > > > > > > > >> > > > > > A couple of points in no particular logical order. > > >> > > > > > > > >> > > > > > (1) HttpCore does not depend on a logging toolkit and does > not > > >> do any > > >> > > > > > logging per default. If it ever encounters an abnormal > > >> situation, > > >> > > > > > instead of logging such an event it throws an exception and > > >> terminates. > > >> > > > > > Quite often this is not desirable, so one can provide an > > >> exception > > >> > > > > > handler that can log using a a logging toolkit of choice or > > >> handle a > > >> > > > > > particular type of exception in an application specific way. > > >> > > > > > > > >> > > > > > If the server did not terminate under with an exception, I > kind > > >> of > > >> > > > > > suspect it was still trying to chew through the load without > > >> > > > > > encountering anything abnormal. Unless you have added a > custom > > >> > > > exception > > >> > > > > > handler. > > >> > > > > > > > >> > > > > > (2) HttpCore does not have a concept of accepted connection > > >> queue. As > > >> > > > > > soon as the listener accepts an incoming connection its > channel > > >> gets > > >> > > > > > immediately added to the underlying i/o selector. No > > >> intermediate > > >> > > > > > queuing takes place. > > >> > > > > > > >> > > > > > > >> > > > > > (3) Given that the connection times out on the client, this > > >> sounds more > > >> > > > > > like a client side issue. Was connection really suck in > > >> > > > > > Socket#socketConnect for 360000? > > >> > > > > > > > >> > > > > > > >> > > > > I'm using httpcore-ab [1] as the client. I have set socket > > >> timeout of > > >> > > > using > > >> > > > > -t 360000, and changed the connection timeout with the patch > > >> shown below. > > >> > > > > But I'm starting to get this error well before 360000 millis > past > > >> the > > >> > > > test > > >> > > > > startup. Therefore, I there is no possibility that the > connection > > >> attempt > > >> > > > > hangs for 360000 millis before printing this exception. I was > > >> thinking > > >> > > > > whether there are upper bounds for these limits that cause my > > >> settings to > > >> > > > > be neglected. > > >> > > > > > > >> > > > > > >> > > > You see. If java.net.ConnectException: Connection timed out is > > >> thrown by > > >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis > elapse, > > >> > > > there must be something wrong with the client side. > > >> > > > > > >> > > > > >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure > that > > >> it > > >> > > doesn't wait 360000 millis before throwing this exception. > > >> > > > > >> > > > >> > > >> Hi Sadeep > > >> > > >> It did turn out to be an issue with the way HttpCore AB initialized > > >> connections. It was basically using an old (Java 1.0) socket > constructor > > >> whose behavior can differ between OS platforms and possibly between > JRE > > >> major releases with regards to connect timeout handling. The connect > > >> timeout value set in the HttpParams simply had no effect and therefore > > >> connections in your tests were timing out before 360000 milliseconds > > >> elapsed (probably using some sort of a system default). > > >> > > >> I committed a fix for the problem > > >> > > >> http://svn.apache.org/viewvc?rev=1238566&view=rev > > >> > > >> Please review > > >> > > > > > > Many thanks for looking into this. I will review this at my earliest > and > > > let you know the outcome. > > > > > > Thanks, > > > Sadeep > > > > > > > > >> > > >> Oleg > > >> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] > > >> For additional commands, e-mail: [email protected] > > >> > > >> > > > > > > > > > -- > > > > > > Sadeep Jayasumana**** > > > > > > Email: [email protected]**** > > > > > > Phone: +94-77-2266507 > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Sadeep Jayasumana**** Email: [email protected]**** Phone: +94-77-2266507
