Hi, On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <[email protected]> wrote:
> On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote: > > Hi, > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <[email protected]> > wrote: > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote: > > > > Hi, > > > > > > > > I have written an HTTP server using HTTPCore-NIO and currently > testing it > > > > with httpcore-ab [1]. I have been seeing very good TPS values so > far. But > > > > when I try to send 5KB HTTP messages with 2500 and concurrency > level, 50 > > > > messages per thread, I'm getting the following exception several > times > > > from > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab > > > output. > > > > No exceptions are displayed from the server side. Hence, I think the > > > server > > > > is rejecting some socket connections because it's already fully > loaded. > > > > > > > > I'm running the server and load generator on two server machines > that are > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB > > > memory > > > > etc. > > > > > > > > I have increased the socket timeout in both httpcore-ab and server > side > > > > ListeningIOReactor to 360000. 16 threads are allocated for the > reactor. > > > > Messages received by the server are processed in a separate worker > pool > > > > which has ~500 threads. I have also increased the open file limit of > my > > > OS > > > > (Debian) to 65535. > > > > > > > > Is there any tuning parameter of HTTPCore that I can used to get rid > of > > > > this exception? Perhaps, something similar to "acceptQueueSize" in > Jetty > > > > [2]? > > > > > > > > > > Hi Sadeep > > > > > > A couple of points in no particular logical order. > > > > > > (1) HttpCore does not depend on a logging toolkit and does not do any > > > logging per default. If it ever encounters an abnormal situation, > > > instead of logging such an event it throws an exception and terminates. > > > Quite often this is not desirable, so one can provide an exception > > > handler that can log using a a logging toolkit of choice or handle a > > > particular type of exception in an application specific way. > > > > > > If the server did not terminate under with an exception, I kind of > > > suspect it was still trying to chew through the load without > > > encountering anything abnormal. Unless you have added a custom > exception > > > handler. > > > > > > (2) HttpCore does not have a concept of accepted connection queue. As > > > soon as the listener accepts an incoming connection its channel gets > > > immediately added to the underlying i/o selector. No intermediate > > > queuing takes place. > > > > > > > (3) Given that the connection times out on the client, this sounds more > > > like a client side issue. Was connection really suck in > > > Socket#socketConnect for 360000? > > > > > > > I'm using httpcore-ab [1] as the client. I have set socket timeout of > using > > -t 360000, and changed the connection timeout with the patch shown below. > > But I'm starting to get this error well before 360000 millis past the > test > > startup. Therefore, I there is no possibility that the connection attempt > > hangs for 360000 millis before printing this exception. I was thinking > > whether there are upper bounds for these limits that cause my settings to > > be neglected. > > > > You see. If java.net.ConnectException: Connection timed out is thrown by > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse, > there must be something wrong with the client side. > Could this be a possible bug in httpcore-ab code? I'm quite sure that it doesn't wait 360000 millis before throwing this exception. Thanks, Sadeep > > Here's the contract of the method: > -------------------------------- > Connects this socket to the server with a specified timeout value. A > timeout of zero is interpreted as an infinite timeout. The connection > will then block until established or an error occurs. > > Parameters: > endpoint - the SocketAddress > timeout - the timeout value to be used in milliseconds. > Throws: > IOException - if an error occurs during the connection > SocketTimeoutException - if timeout expires before connecting > IllegalBlockingModeException - if this socket has an associated > channel, and the channel is in non-blocking mode > IllegalArgumentException - if endpoint is null or is a > SocketAddress subclass not supported by this socket > -------------------------------- > > ConnectException is a subclass of SocketTimeoutException. My > understanding of the contract is that the exception cannot be thrown > unless the timeout in milliseconds fully elapses. > > Oleg > > > --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision > > 1205292) > > +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working > > copy) > > @@ -227,7 +227,8 @@ > > .setParameter(HttpProtocolParams.USER_AGENT, > "HttpCore-AB/1.1") > > .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE, > > useExpectContinue) > > > > .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false) > > - .setIntParameter(HttpConnectionParams.SO_TIMEOUT, > > socketTimeout); > > + .setIntParameter(HttpConnectionParams.SO_TIMEOUT, > > socketTimeout) > > + .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, > 360 > > * 1000); > > return params; > > } > > } > > > > > > > > > > (4) Quite often it is useful to see what it really going on on the I/O > > > reactor level. You might want to add I/O event and wire logging by > > > copying logging components from the contrib package [1] > > > > > > > I could do this. But will wire logging really help at this concurrency > > level and messages/thread? > > > > [1] > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab > > > > Thanks, > > Sadeep > > > > > > > > I hope this helps somewhat. In any way I can say I am quite sure > > > HttpCore does not reject incoming connections no matter what. > > > > > > > Oleg > > > > > > [1] > > > > > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/ > > > > > > > [1] > > > > > > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab > > > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors > > > > [3] > > > > java.net.ConnectException: Connection timed out > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > > > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > > > at java.net.Socket.connect(Socket.java:529) > > > > at java.net.Socket.connect(Socket.java:478) > > > > at java.net.Socket.<init>(Socket.java:375) > > > > at java.net.Socket.<init>(Socket.java:189) > > > > at > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207) > > > > at > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > java.net.ConnectException: Connection timed out > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > > > at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > > > at java.net.Socket.connect(Socket.java:529) > > > > at java.net.Socket.connect(Socket.java:478) > > > > at java.net.Socket.<init>(Socket.java:375) > > > > at java.net.Socket.<init>(Socket.java:189) > > > > at > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207) > > > > at > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > Thanks & Regards, > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Sadeep Jayasumana**** Email: [email protected]**** Phone: +94-77-2266507
