Hi folks,
coincidentally I am also having problems with large number of concurrent
connections to Tomcat
getting errors when we try and connect many clients concurrently to Tomcat.
Does anyone have experience of this sort of Tomcat usage or should we be
looking at a different servlet container for heavy loads.
The setup is as follows
Tomcat 3.2.2. on WinNT SP 6 running as a standalone servlet container.
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)
We have a test environment where a number (2-4) of other machines, with
multi-threaded clients (30-1000 threads) bombard the servlet in an effort
to test the servlet performance. This leads to a number of problems
1) First of all the mutli-threaded test clients (also written in Java)
occasionally throw a NoClassDefFoundError when the following line of code
is executed
PrintWriter out = new PrintWriter(conn.getOutputStream());
The class that is not found is a Socket class (sorry I don't have the trace
at the moment) is not found. This happens when the number of threads on the
client is high (c. 1000). Obviously it is not a Tomcat problem so I
shouldn't even mention it. It is easily worked around by catching for the
error and retrying.
It looks slightly similar to a problem in JKD 1.2 where the Calender class
would load a ResourceBundle containing Locale info (in particular Date
format info) and would report a MissingResourceBundle exception
occasionally with a high number of threads creating Calenders
simultaneously. A chain of resources are created and loaded into a
HashTable during a call to load the resource, but some of the resources may
be garbage collected during recursive calls to ResourceBundle.getObject()
resulting in a MissingResourceException. Since garbage collection is not
very frequent the chances of this occurring during a resource access are
quite low. I only mention it since it has some surface similarities with my
problem. For more info look here
http://developer.java.sun.com/developer/bugParade/bugs/4335714.html
2) The second problem may be an actual Tomcat problem.
It is caused by the same line of code and is an IOException saying that the
connection has been refused. Even when we increase various parameters in
the server.xml file -
<Parameter name="backlog" value="4096" />
<!-- Make the threads work better for multiple connections. -->
<Parameter name="max_threads" value="4096"/>
<Parameter name="max_spare_threads" value="256"/>
<Parameter name="min_spare_threads" value="128">
to ensure that Tomcat will accept more connections it does not seem to
work. This problem has been demonstrated to be independent of out servlet
and appears when accessing a resource such as a JSP file. Again this is not
a critical problem since we can catch the exception and retry the
connection but it is puzzling as to why the high backlog figure doesn't
ensure that the socket connection attempts are not queued. Again sorry that
I don't have a trace to hand at the moment.
3) We have more serious problems on the Tomcat side, after some while (a
couple of hours) with a high number (1000 -1500) of clients continuously
connecting we get errors in the tomcat logs
"2001-08-07 09:33:47 - Ctx( /certresolver ): IOException in: R(
/certresolver+ /certs + null) socket write error (code=10055)"
A quick search for code=10055 reveals it to be an error code thrown by
winsock, however it covers a couple of different situations including
trying to increase the timeout on a socket. The one that appears most
appropriate to this problem is a bug report at sun
http://developer.java.sun.com/developer/bugParade/bugs/4474929.html
"Embedded in the exception's message string is code 10055. Under Win32,
this error is WSAENOBUFS which means the TCP/IP stack has temporarily run
out of buffer space and cannot complete the write. Microsoft has verified
this as a bug in the WinSock TCP/IP stack (see the MSDN article ID:
Q201213). This would be a recoverable error if not for the way the Java
native code handles the error. The recovery would simply be to wait for
the stack buffers to empty and try sending the same data again at a later
time."
It could possibly be related to this error but without more information in
the exception we can't tell. Has anyone ever experienced something like this?
4) The final problem is that Tomcat reports is an OutOfMemoryException
after continuous usage with a high number of clients. Increasing the memory
available to the Tomcat process (using -Xmx500m) does not make the problem
go away, the taskmanager reports that the Tomcat process is only using a
fraction of its allocated memory. Could this be a related to the previous
discussion on upper limits for blocking I/O connections per process
Many thanks if you can help in any way.
Eamonn
>Well I think you're completely nuts :) For 10,000 concurrent requests for
>*dynamic* data (or else you'd just be using Apache.. right?), you'd need a
>server farm full of fancy equipment, a load-balancing server, and stuff that
>even my wildly ignorant speculations can't conceive.
>
>Besides, I don't think any single server with simple thread-per-request,
>blocking IO model can handle that sort of load. A smart fellow called Matt
>Welsh did his PhD thesis on highly concurrent server apps, and the results of
>his thinking are here:
>
>http://www.cs.berkeley.edu/~mdw/proj/sandstorm/
>
>In particular, the paper "The Staged Event-Driven Architecture for Highly
>Concurrent Servers." is well worth reading.
>
>--Jeff
>(who thinks 50 concurrent users is pushing things)
>
>
>On Mon, Aug 13, 2001 at 01:54:28PM +0530, Santosh Pasi wrote:
> > Hi Jeff,
> >
> > Thanks for reply.
> > I am looking around 10,000 to 12,000 concurrent connections.
>[snip]
-----------------------------------------------------------------------------
Baltimore Technologies plc will not be liable for direct, special, indirect
or consequential damages arising from alteration of the contents of this
message by a third party or as a result of any virus being passed on.
In addition, certain Marketing collateral may be added from time to time to
promote Baltimore Technologies products, services, Global e-Security or
appearance at trade shows and conferences.
This footnote confirms that this email message has been swept by
Baltimore MIMEsweeper for Content Security threats, including
computer viruses.
http://www.baltimore.com
