Hi!
it will be best if I describe our configuration at first:
We are running Solaris 7, Tomcat 3.2.2 b4 (beta 4) + AJP12 Connector (Tomcat
configured with max of 500 threads), JVM: Sun 1.3.0_02 Hot Spot Native
Threads, Apache 1.3.19 (max of 1024 child processes)
We have had substantial traffic on our server (we have 7 servers load
balanced with the configuration specified above)
We found that if the incoming HTTP request has content-length missing from
the header, and the request gets channeled
via the AJP12 connector to the Tomcat and the servlet, it appears that all
subsequent request are causing AJP12 connections
to raise (All in ESTABLISHED mode) and Tomcat refuses to process any more
request , eventually eating up great deal of CPU time, we needed to do
restart and both Apache and the Tomcat to recover from this condition.
We have then decided to bypass all requests with the content-length missing
in the Ajp12ConnectionHandler.java
int contentLength = reqA.getMimeHeaders().getIntHeader("content-length");
if (contentLength != -1) {
BufferedServletInputStream sis =
(BufferedServletInputStream)reqA.getInputStream();
sis.setLimit(contentLength);
}
+ else {
+ resA.finish();
+ socket.close();
+ return;
+ }
contextM.service( reqA, resA );
//resA.finish(); // is part of contextM !
socket.close();
} catch (Exception e) {
// XXX
// this isn't what we want, we want to log the problem somehow
System.out.println("HANDLER THREAD PROBLEM: " + e);
e.printStackTrace();
}
Note that in the above socket.close() is not executed when an exception is
thrown in the service method
we are considering adding a "finally" clause to handle such a case: i.e
contextM.service( reqA, resA );
//resA.finish(); // is part of contextM !
- socket.close();
} catch (Exception e) {
// XXX
// this isn't what we want, we want to log the problem somehow
System.out.println("HANDLER THREAD PROBLEM: " + e);
e.printStackTrace();
}
+ } finally() {
+ if (socket != null)
+ socket.close();
+ }
----------------------------------------------------------------------------
-----------------------------
I believe that potential offending code could in the servlet application
that tries to read input stream "request.getInputStream() without the
content-length
following is what the servlet code looks like:
private int readInputStream(HttpServletRequest request) {
int clientDayOfYear = 0;
try {
InputStreamReader isr=new
InputStreamReader(request.getInputStream());
BufferedReader br=new BufferedReader(isr);
clientDayOfYear = new
Integer(br.readLine().toString()).intValue();
isr.close();
isr=null;
br.close();
br=null;
} catch (IOException e) {
clientDayOfYear=0;
}
return clientDayOfYear;
}
We know that our change to the Tomcat code is temporary at best (it allows
us to go on!) I am looking for the feedback
on the "quality" of our code change as well as maybe to a deeper reasons why
content-length missing from the header
together with request.getInputStream() would cause Apache/AJP12 to have
connections open (All in ESTABLISHED mode) and
eventually eat up all available CPU resources
Thanks for any feedback/help!
