Http connection listener terminates when request executor full
--------------------------------------------------------------
Key: AXIS2-4800
URL: https://issues.apache.org/jira/browse/AXIS2-4800
Project: Axis2
Issue Type: Bug
Components: transports
Affects Versions: 1.5, 1.5.1, 1.6, 1.5.2, nightly
Environment: Any
Reporter: Chuck Williams
The DefaultConnectionListener of the built-in http server invokes the failure
apparatus in the configured ConnectionListenerFailureHandler when exceptions
arise handing off a new request connection to an HttpServiceProcessor. The
built-in failure apparatus in DefaultConnectionListenerFailureHandler retries
(default 10 times) and then gives up, after which the DefaultConnectionListener
shuts down. This permanently closes the port, rendering the web service
unreachable. One can replace the DefaultConnectionListenerFailureHandler
easily by overriding HttpFactory.newRequestConnectionListener(), however the
failure processing methods do not have access to the request connection and
thus cannot take any action that requires sending a response back to the client.
A particularly important case arises when the http request executor's thread
pool and backing queue are both full. In this case the attempt to launch a
thread to process the new http request gets RejectedExecutionException. The
server should return an http 503 Service Unavailable, since it has already
accessed the new connection and so cannot refuse it. This is impossible to
achieve with the existing failure apparatus because it can't send a response.
The result is that the when the web service becomes overloaded the connection
listener permanently shuts down!
The failure apparatus is actually a patch of mine from a few years ago. At the
time any failure would immediately shut down the connection listener and I
added some simple processing for the case of an intermittent issue. The new
case of RejectedExecutionException is clearly an important one and I've
prepared a new patch (to be attached next) that fixes this by sending 503
Service Unavailable whenever the request executor cannot accept a new request
after the connection listener has already accepted the request connection.
I would have committed this, but since I've been inactive in axis2 development
for some time that seemed unwise. An active developer should probably first
weigh in. I have tested the patch successfully in an application that requires
it.
More should be done here. Failure cases in the connection listener are
important as the consequences of improper handling are dire, leaving your web
service unreachable. Probably there are additional cases that should be built
in, and the extensible failure apparatus should be generalized so that it can
send responses whenever a request has already been accepted but the normal
processing fails back to the listener.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
