https://bz.apache.org/bugzilla/show_bug.cgi?id=70366

            Bug ID: 70366
           Summary: HTTP/2: Content-Length lookup in Stream.receivedData()
                    races with RemoteIpValve header removal and can return
                    another header's value
           Product: Tomcat 11
           Version: unspecified
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Connectors
          Assignee: [email protected]
          Reporter: [email protected]
  Target Milestone: -------

With HTTP/2 and RemoteIpValve, a request that has a body and a forwarding
header is occasionally reset with PROTOCOL_ERROR even though its
Content-Length is valid. Tomcat reads the value of a different header (the
one that follows Content-Length) as the content length.

Tested with 11.0.29, the current release. The methods involved are the same
on 11.0.x (e7fa79abe5) and main (9e13426ab9) as of 2026-11-21.

This looks like the same problem as bug 69479 (Tomcat 9), which was resolved
after the test case stopped failing on 9.0.100. The stack traces match, see
"Relation to bug 69479" below.


Symptom
-------
Log from HeaderRaceProbe (h2c, loopback, RemoteIpValve with
default settings, POST with a 128 byte body, headers Authorization,
X-Forwarded-For, Content-Type, Content-Length, User-Agent):

INFO: Connection [0], Stream [176787] Closed due to error
org.apache.coyote.http2.StreamException: Connection [0], Stream [176787], The
content length header was invalid
        at org.apache.coyote.http2.Stream.getContentLengthLong(Stream.java:816)
        at org.apache.coyote.http2.Stream.receivedData(Stream.java:789)
        at
org.apache.coyote.http2.Http2UpgradeHandler.startRequestBodyFrame(Http2UpgradeHandler.java:1773)
        at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.startRequestBodyFrame(Http2AsyncUpgradeHandler.java:46)
        at
org.apache.coyote.http2.Http2Parser.readDataFrame(Http2Parser.java:184)
        at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:263)
        ...
Caused by: java.lang.NumberFormatException: For input string:
"Java-http-client/25.0.2"
        at java.base/java.lang.Long.parseLong(Long.java:594)
        at
org.apache.tomcat.util.buf.MessageBytes.getLong(MessageBytes.java:681)
        at org.apache.coyote.Request.getContentLengthLong(Request.java:686)
        at org.apache.coyote.http2.Stream.getContentLengthLong(Stream.java:812)

"Java-http-client/25.0.2" is the User-Agent value of that request. The
client sees an IOException for the stream.


Analysis
--------
Two threads use the same MimeHeaders instance without synchronisation.

Connection thread, on the first DATA frame of the stream:
  Http2Parser.readDataFrame()
  -> Http2UpgradeHandler.startRequestBodyFrame()
  -> Stream.receivedData()                      (Stream.java:789)
  -> Request.getContentLengthLong()             (Request.java:680)
  -> MimeHeaders.getUniqueValue("content-length")  (MimeHeaders.java:384)

Request.getContentLengthLong() only skips the search once contentLength has
been cached as a value > -1, so a DATA frame that arrives before the value
is cached searches the header array.

Stream processing thread, started once the headers are complete:
  StreamProcessor.service()
  -> CoyoteAdapter.service()
  -> RemoteIpValve.invoke()
  -> MimeHeaders.removeHeader(String)           (RemoteIpValve.java:664, 671;
                                                 again at 758, 763 when the
                                                 original values are restored)

MimeHeaders.removeHeader(int) recycles the field, shifts the following
entries down with System.arraycopy() and decrements count
(MimeHeaders.java:428). getUniqueValue() reads headers[i].getName() and then
headers[i].getValue() as two separate array reads. If the shift happens
between them, the name comes from Content-Length and the value from the next
header. If the shift happens before the reader reaches Content-Length, the
reader can skip it and return null.

I see both outcomes with the attached HeaderMutationProbe:

  - NumberFormatException with the following header's value (the failure
    above). In 11.0.29 this is reported as a stream reset.
  - getContentLengthLong() returns -1 for a request that has
    Content-Length: 128. This one is silent. The value is not cached, so the
    next DATA frame searches again.

I have not observed a wrong numeric value being accepted, but from the code
it looks possible if the header after Content-Length has a numeric value.


How to reproduce
----------------
The two test cases are attached. They are also available as a small Gradle
project, with complete logs, at
https://github.com/waileong/tomcat-http2-content-length-race
The only dependency is tomcat-embed-core 11.0.29. I used JDK 25.

1. HeaderMutationProbe.java (attached). No network. One thread calls
   Request.getContentLengthLong() while another removes a preceding header,
   100,000 rounds, headers x-forwarded-for / content-length / user-agent.

     ./gradlew mutationProbe

   Windows 11, JDK 25.0.3: every run failed, with 1 to 9 invalid reads per
   100,000 rounds. Output of one run:

     8 x returned -1
     99991 x returned 128
     1 x threw java.lang.NumberFormatException: For input string:
"Java-http-client/synthetic"

2. HeaderRaceProbe.java (attached). Embedded Tomcat on loopback with h2c,
   RemoteIpValve added to the engine, one java.net.http HTTP/2 client with
   32 concurrent senders. First argument enables the valve, second is the
   duration in seconds, the optional third selects the connector executor
   (virtual, the default, or platform for Tomcat's default thread pool).

     ./gradlew raceProbe --args="true 120"

   The probe stops at the first failed request. Linux container
   (ghcr.io/graalvm/jdk-community:25, JDK 25.0.2, 4 CPUs, 2 GiB).

   120 s runs:

     valve  executor  outcome
     yes    virtual   failed after 36,589 requests
     yes    virtual   2,714,637 requests, no failure
     yes    platform  six runs of 1,074,896 to 1,618,026 requests, no failure
     no     virtual   2,900,517 requests, no failure

   5 s runs, valve enabled, platform executor: 2 of 20 runs failed, after
   545 and 1,373 requests. The other 18 completed 29,480 to 32,387 requests.

   Each failure has the stack trace above. So with a virtual thread executor
   the failure also appears in a warmed-up server, and with the default
   thread pool I have only seen it in the first seconds after start-up. The
   end-to-end case is timing dependent and a run can pass. I have only seen
   it fail on Linux; runs on Windows 11 have not failed.


Relation to bug 69479
---------------------
Bug 69479 comment 0 has the same reader trace on 9.0.65:
  MessageBytes.getLong <- Request.getContentLengthLong <- Stream.receivedData
with another header's value ("application/json; charset=UTF-8") as the
input string. Comment 8 has the writer trace:
  MimeHeaders.removeHeader <- RemoteIpValve.invoke <- CoyoteAdapter.service
  <- StreamProcessor.service
That bug was resolved because the reporter's test case no longer failed on
9.0.100. The two code paths are still present in 11.0.29, so I think the
later changes altered the timing rather than removed the race.


Possible direction
------------------
Resolve the content length on the connection thread when the header block
is complete, before the stream is passed to a StreamProcessor, and have
receivedData() use that value instead of searching MimeHeaders. I have not
tested a patch.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to