https://bz.apache.org/bugzilla/show_bug.cgi?id=70290
Bug ID: 70290
Summary: Async in-progress count leaks when onError() calls
dispatch() after a client disconnect during initial
service
Product: Tomcat 11
Version: 11.0.26
Hardware: Macintosh
Status: NEW
Severity: normal
Priority: P2
Component: Catalina
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: -------
Created attachment 40238
--> https://bz.apache.org/bugzilla/attachment.cgi?id=40238&action=edit
Test case that reproduces the bug
Overview
--------
When a client drops the connection during the initial asynchronous service
call, and an AsyncListener.onError() handler responds by calling
AsyncContext.dispatch(), the request can be recycled while the async state
machine is still in the MUST_DISPATCH state. As a result
AsyncListener.onComplete() is never called and
StandardContext.inProgressAsyncCount is never decremented. The count leaks by
one per affected request. Because the container waits for in-progress async
requests when stopping, a non-zero residual count can prevent a graceful
shutdown from completing.
This is related to BZ 68227 (resolved), which added onComplete() handling to
the DISPATCHING branch of the async state machine. This report covers the
MUST_DISPATCH path that is reached when the dispatch is requested during the
initial service call, which that fix did not address.
Version tested
both Linux (Ubuntu, OpenJDK 25) and macOS (OpenJDK 27). The affected code is
present unchanged in 11.0.x.
Configuration
-------------
Default configuration, NIO connector. No special settings are required.
Behavior / root cause
---------------------
Two locations participate:
1. CoyoteAdapter.service() forces an ASYNC_POST_PROCESS during its cleanup
only when request.isAsyncCompleting() is true. The sibling asyncDispatch()
path covers both isAsyncCompleting() and isAsyncDispatching(). When the
connection is being closed after the initial service call, the processor
does not run asyncPostProcess() itself (it returns CLOSED), so the forced
call in service() is the only opportunity to resolve the async lifecycle.
Because the request is dispatching rather than completing, that call is
skipped and the MUST_DISPATCH state is abandoned.
2. AsyncStateMachine.asyncPostProcess() handles MUST_DISPATCH by transitioning
to DISPATCHING unconditionally. It does not fire onComplete() or decrement
the in-progress count when a connection error means the requested dispatch
can no longer run. By contrast, the DISPATCHING and MUST_COMPLETE branches
already check isIoAllowed() and complete the lifecycle in that situation.
Steps to reproduce
------------------
See the attached JUnit test, TestAsyncResetInProgressCount.java, which extends
TomcatBaseTest. It:
* registers an async servlet that starts async, sets a large content length,
and writes the response body from an application worker thread;
* adds an AsyncListener whose onError() is coordinated with the worker (using
latches) so that the worker's AsyncContext.dispatch() overlaps onError(),
mirroring how Spring's StreamingResponseBody returns to the container after
a client disconnect;
* uses a plain socket client that reads part of the response and then closes
with SO_LINGER = 0 to send a TCP RST;
* widens the initial service() window with a short filter hold so the write
error and the dispatch land while the async state is still STARTING, which
is the specific initial-service schedule described above;
* asserts that onComplete() fires exactly once and that
getInProgressAsyncCount() returns to its baseline on every round.
Without a fix the test fails on every round: onComplete() is not called and
inProgressAsyncCount grows monotonically (for example from 0 to 20 over 20
rounds, leaving a residual of 20). A natural, uncoordinated variant with the
hold set to 0 is included as a control and passes, because the error then
lands in the already-handled STARTED state. This was observed consistently on
both Linux and macOS.
Expected behavior
-----------------
A dispatch requested from onError() must be honored. When the connection is
still usable the dispatch should run; when it is not, the async lifecycle must
still complete so that onComplete() is called and the in-progress count is
balanced. Simply restoring the count without resolving the requested dispatch
would be insufficient (see BZ 68227 comment 5).
A patch is provided as a GitHub pull request. It extends the
CoyoteAdapter.service() cleanup to also force ASYNC_POST_PROCESS for a
dispatching async request, and completes the lifecycle from the MUST_DISPATCH
branch of asyncPostProcess() when I/O is no longer allowed, mirroring the
existing DISPATCHING branch.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
