On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote: > Hi, > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <[email protected]> wrote: > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote: > > > Hi, > > > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <[email protected]> > > wrote: > > > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote: > > > > > Hi, > > > > > > > > > > I have written an HTTP server using HTTPCore-NIO and currently > > testing it > > > > > with httpcore-ab [1]. I have been seeing very good TPS values so > > far. But > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency > > level, 50 > > > > > messages per thread, I'm getting the following exception several > > times > > > > from > > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab > > > > output. > > > > > No exceptions are displayed from the server side. Hence, I think the > > > > server > > > > > is rejecting some socket connections because it's already fully > > loaded. > > > > > > > > > > I'm running the server and load generator on two server machines > > that are > > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB > > > > memory > > > > > etc. > > > > > > > > > > I have increased the socket timeout in both httpcore-ab and server > > side > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the > > reactor. > > > > > Messages received by the server are processed in a separate worker > > pool > > > > > which has ~500 threads. I have also increased the open file limit of > > my > > > > OS > > > > > (Debian) to 65535. > > > > > > > > > > Is there any tuning parameter of HTTPCore that I can used to get rid > > of > > > > > this exception? Perhaps, something similar to "acceptQueueSize" in > > Jetty > > > > > [2]? > > > > > > > > > > > > > Hi Sadeep > > > > > > > > A couple of points in no particular logical order. > > > > > > > > (1) HttpCore does not depend on a logging toolkit and does not do any > > > > logging per default. If it ever encounters an abnormal situation, > > > > instead of logging such an event it throws an exception and terminates. > > > > Quite often this is not desirable, so one can provide an exception > > > > handler that can log using a a logging toolkit of choice or handle a > > > > particular type of exception in an application specific way. > > > > > > > > If the server did not terminate under with an exception, I kind of > > > > suspect it was still trying to chew through the load without > > > > encountering anything abnormal. Unless you have added a custom > > exception > > > > handler. > > > > > > > > (2) HttpCore does not have a concept of accepted connection queue. As > > > > soon as the listener accepts an incoming connection its channel gets > > > > immediately added to the underlying i/o selector. No intermediate > > > > queuing takes place. > > > > > > > > > > (3) Given that the connection times out on the client, this sounds more > > > > like a client side issue. Was connection really suck in > > > > Socket#socketConnect for 360000? > > > > > > > > > > I'm using httpcore-ab [1] as the client. I have set socket timeout of > > using > > > -t 360000, and changed the connection timeout with the patch shown below. > > > But I'm starting to get this error well before 360000 millis past the > > test > > > startup. Therefore, I there is no possibility that the connection attempt > > > hangs for 360000 millis before printing this exception. I was thinking > > > whether there are upper bounds for these limits that cause my settings to > > > be neglected. > > > > > > > You see. If java.net.ConnectException: Connection timed out is thrown by > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse, > > there must be something wrong with the client side. > > > > Could this be a possible bug in httpcore-ab code? I'm quite sure that it > doesn't wait 360000 millis before throwing this exception. >
Of course, it is not impossible. However, if you take look at the exception stack trace, the exception originates from native part of JRE code, which is responsible for low level TCP/IP functionality. Have you tried using an infinite timeout (no timeout)? Another thing can think of is trying a different client side JRE. Oleg > Thanks, > Sadeep > > > > > > Here's the contract of the method: > > -------------------------------- > > Connects this socket to the server with a specified timeout value. A > > timeout of zero is interpreted as an infinite timeout. The connection > > will then block until established or an error occurs. > > > > Parameters: > > endpoint - the SocketAddress > > timeout - the timeout value to be used in milliseconds. > > Throws: > > IOException - if an error occurs during the connection > > SocketTimeoutException - if timeout expires before connecting > > IllegalBlockingModeException - if this socket has an associated > > channel, and the channel is in non-blocking mode > > IllegalArgumentException - if endpoint is null or is a > > SocketAddress subclass not supported by this socket > > -------------------------------- > > > > ConnectException is a subclass of SocketTimeoutException. My > > understanding of the contract is that the exception cannot be thrown > > unless the timeout in milliseconds fully elapses. > > > > Oleg > > > > > --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision > > > 1205292) > > > +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working > > > copy) > > > @@ -227,7 +227,8 @@ > > > .setParameter(HttpProtocolParams.USER_AGENT, > > "HttpCore-AB/1.1") > > > .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE, > > > useExpectContinue) > > > > > > .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false) > > > - .setIntParameter(HttpConnectionParams.SO_TIMEOUT, > > > socketTimeout); > > > + .setIntParameter(HttpConnectionParams.SO_TIMEOUT, > > > socketTimeout) > > > + .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, > > 360 > > > * 1000); > > > return params; > > > } > > > } > > > > > > > > > > > > > > (4) Quite often it is useful to see what it really going on on the I/O > > > > reactor level. You might want to add I/O event and wire logging by > > > > copying logging components from the contrib package [1] > > > > > > > > > > I could do this. But will wire logging really help at this concurrency > > > level and messages/thread? > > > > > > [1] > > > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab > > > > > > Thanks, > > > Sadeep > > > > > > > > > > > I hope this helps somewhat. In any way I can say I am quite sure > > > > HttpCore does not reject incoming connections no matter what. > > > > > > > > > > Oleg > > > > > > > > [1] > > > > > > > > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/ > > > > > > > > > [1] > > > > > > > > > > > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab > > > > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors > > > > > [3] > > > > > java.net.ConnectException: Connection timed out > > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > > > > at > > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > > > > at java.net.Socket.connect(Socket.java:529) > > > > > at java.net.Socket.connect(Socket.java:478) > > > > > at java.net.Socket.<init>(Socket.java:375) > > > > > at java.net.Socket.<init>(Socket.java:189) > > > > > at > > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207) > > > > > at > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > at > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > at java.lang.Thread.run(Thread.java:662) > > > > > java.net.ConnectException: Connection timed out > > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) > > > > > at > > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) > > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) > > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > > > > > at java.net.Socket.connect(Socket.java:529) > > > > > at java.net.Socket.connect(Socket.java:478) > > > > > at java.net.Socket.<init>(Socket.java:375) > > > > > at java.net.Socket.<init>(Socket.java:189) > > > > > at > > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207) > > > > > at > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > at > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > > Thanks & Regards, > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
