https://bz.apache.org/bugzilla/show_bug.cgi?id=70366
Bug ID: 70366
Summary: HTTP/2: Content-Length lookup in Stream.receivedData()
races with RemoteIpValve header removal and can return
another header's value
Product: Tomcat 11
Version: unspecified
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: -------
With HTTP/2 and RemoteIpValve, a request that has a body and a forwarding
header is occasionally reset with PROTOCOL_ERROR even though its
Content-Length is valid. Tomcat reads the value of a different header (the
one that follows Content-Length) as the content length.
Tested with 11.0.29, the current release. The methods involved are the same
on 11.0.x (e7fa79abe5) and main (9e13426ab9) as of 2026-11-21.
This looks like the same problem as bug 69479 (Tomcat 9), which was resolved
after the test case stopped failing on 9.0.100. The stack traces match, see
"Relation to bug 69479" below.
Symptom
-------
Log from HeaderRaceProbe (h2c, loopback, RemoteIpValve with
default settings, POST with a 128 byte body, headers Authorization,
X-Forwarded-For, Content-Type, Content-Length, User-Agent):
INFO: Connection [0], Stream [176787] Closed due to error
org.apache.coyote.http2.StreamException: Connection [0], Stream [176787], The
content length header was invalid
at org.apache.coyote.http2.Stream.getContentLengthLong(Stream.java:816)
at org.apache.coyote.http2.Stream.receivedData(Stream.java:789)
at
org.apache.coyote.http2.Http2UpgradeHandler.startRequestBodyFrame(Http2UpgradeHandler.java:1773)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.startRequestBodyFrame(Http2AsyncUpgradeHandler.java:46)
at
org.apache.coyote.http2.Http2Parser.readDataFrame(Http2Parser.java:184)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:263)
...
Caused by: java.lang.NumberFormatException: For input string:
"Java-http-client/25.0.2"
at java.base/java.lang.Long.parseLong(Long.java:594)
at
org.apache.tomcat.util.buf.MessageBytes.getLong(MessageBytes.java:681)
at org.apache.coyote.Request.getContentLengthLong(Request.java:686)
at org.apache.coyote.http2.Stream.getContentLengthLong(Stream.java:812)
"Java-http-client/25.0.2" is the User-Agent value of that request. The
client sees an IOException for the stream.
Analysis
--------
Two threads use the same MimeHeaders instance without synchronisation.
Connection thread, on the first DATA frame of the stream:
Http2Parser.readDataFrame()
-> Http2UpgradeHandler.startRequestBodyFrame()
-> Stream.receivedData() (Stream.java:789)
-> Request.getContentLengthLong() (Request.java:680)
-> MimeHeaders.getUniqueValue("content-length") (MimeHeaders.java:384)
Request.getContentLengthLong() only skips the search once contentLength has
been cached as a value > -1, so a DATA frame that arrives before the value
is cached searches the header array.
Stream processing thread, started once the headers are complete:
StreamProcessor.service()
-> CoyoteAdapter.service()
-> RemoteIpValve.invoke()
-> MimeHeaders.removeHeader(String) (RemoteIpValve.java:664, 671;
again at 758, 763 when the
original values are restored)
MimeHeaders.removeHeader(int) recycles the field, shifts the following
entries down with System.arraycopy() and decrements count
(MimeHeaders.java:428). getUniqueValue() reads headers[i].getName() and then
headers[i].getValue() as two separate array reads. If the shift happens
between them, the name comes from Content-Length and the value from the next
header. If the shift happens before the reader reaches Content-Length, the
reader can skip it and return null.
I see both outcomes with the attached HeaderMutationProbe:
- NumberFormatException with the following header's value (the failure
above). In 11.0.29 this is reported as a stream reset.
- getContentLengthLong() returns -1 for a request that has
Content-Length: 128. This one is silent. The value is not cached, so the
next DATA frame searches again.
I have not observed a wrong numeric value being accepted, but from the code
it looks possible if the header after Content-Length has a numeric value.
How to reproduce
----------------
The two test cases are attached. They are also available as a small Gradle
project, with complete logs, at
https://github.com/waileong/tomcat-http2-content-length-race
The only dependency is tomcat-embed-core 11.0.29. I used JDK 25.
1. HeaderMutationProbe.java (attached). No network. One thread calls
Request.getContentLengthLong() while another removes a preceding header,
100,000 rounds, headers x-forwarded-for / content-length / user-agent.
./gradlew mutationProbe
Windows 11, JDK 25.0.3: every run failed, with 1 to 9 invalid reads per
100,000 rounds. Output of one run:
8 x returned -1
99991 x returned 128
1 x threw java.lang.NumberFormatException: For input string:
"Java-http-client/synthetic"
2. HeaderRaceProbe.java (attached). Embedded Tomcat on loopback with h2c,
RemoteIpValve added to the engine, one java.net.http HTTP/2 client with
32 concurrent senders. First argument enables the valve, second is the
duration in seconds, the optional third selects the connector executor
(virtual, the default, or platform for Tomcat's default thread pool).
./gradlew raceProbe --args="true 120"
The probe stops at the first failed request. Linux container
(ghcr.io/graalvm/jdk-community:25, JDK 25.0.2, 4 CPUs, 2 GiB).
120 s runs:
valve executor outcome
yes virtual failed after 36,589 requests
yes virtual 2,714,637 requests, no failure
yes platform six runs of 1,074,896 to 1,618,026 requests, no failure
no virtual 2,900,517 requests, no failure
5 s runs, valve enabled, platform executor: 2 of 20 runs failed, after
545 and 1,373 requests. The other 18 completed 29,480 to 32,387 requests.
Each failure has the stack trace above. So with a virtual thread executor
the failure also appears in a warmed-up server, and with the default
thread pool I have only seen it in the first seconds after start-up. The
end-to-end case is timing dependent and a run can pass. I have only seen
it fail on Linux; runs on Windows 11 have not failed.
Relation to bug 69479
---------------------
Bug 69479 comment 0 has the same reader trace on 9.0.65:
MessageBytes.getLong <- Request.getContentLengthLong <- Stream.receivedData
with another header's value ("application/json; charset=UTF-8") as the
input string. Comment 8 has the writer trace:
MimeHeaders.removeHeader <- RemoteIpValve.invoke <- CoyoteAdapter.service
<- StreamProcessor.service
That bug was resolved because the reporter's test case no longer failed on
9.0.100. The two code paths are still present in 11.0.29, so I think the
later changes altered the timing rather than removed the race.
Possible direction
------------------
Resolve the content length on the connection thread when the header block
is complete, before the stream is passed to a StreamProcessor, and have
receivedData() use that value instead of searching MimeHeaders. I have not
tested a patch.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]