On 09/01/2025 14:19, Mark Thomas wrote:
On 03/01/2025 07:44, Mark Thomas wrote:
Arjan,
This is the right place to ask that question. Taking a look at this is
on my TODO list. Between addressing CVE-2024-50379 and CVE-2024-56337
and the holiday season I haven't got to it yet. I expect to look at it
before the next release (it isn't the only issue on my TODO list).
I've made some progress. I understand what is going on and I have a fix
that is working well in testing so far. I have some more testing to do.
If that goes well I expect to commit a fix later today for the next
round of releases.
Just a quick follow-up on this. One of the changes is quite invasive so
we are going to roll it out slowly just to be safe. The fix will be in
the next 11.0.x release and then back-ported to 10.1.x and 9.0.x at
least one release later.
Using discardRequestsAndResponses continues to be a viable workaround.
Note that with this fix in place, each request/response pair on the
stack will still require ~60k.
Mark
Mark
Mark
On 02/01/2025 14:06, Arjan van IJzendoorn wrote:
Hello,
Am I asking questions in the wrong place or in the wrong way? I was
hoping to hear from someone here. Please point me in the right
direction if this is not the place.
This "memory leak" may happen to others, too.
Thanks,
Arjan
________________________________
From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com.INVALID>
Sent: Thursday, December 19, 2024 15:16
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Re: Excessive memory usage for HTTP/2 requests
Hello,
In my initial message, I tried embedding an image but that did not
work. Here is a link to a publicly hosted image instead:
https://ibb.co/4KFXgdj
In the memory analyzer screenshot, you can see the
'recycledRequestAndResponses' stack taking up 288MB, which is more
than half of the total memory usage.
Since the previous message, our service has been running
'discardRequestsAndResponses' set to true and memory usage has been
stable. Is that option there to stay? The comment in the code of
Http2Protocol suggests that using the stack improves performance so I
am a little afraid that the option to turn this optimization off may
disappear in the future.
We are still not sure why Request objects are so large in our case.
Kind regards, Arjan
________________________________
From: Arjan van IJzendoorn <arjan.van.ijzendo...@blockbax.com>
Sent: Tuesday, December 17, 2024 13:41
To: users@tomcat.apache.org <users@tomcat.apache.org>
Subject: Excessive memory usage for HTTP/2 requests
Hello Tomcat friends,
Our Spring Boot 3.4.0 application uses Tomcat 10.1.33. Recently we
started seeing the memory usage of our app grow and after three days
it would reach its memory limit and crash.
We investigated heap dumps and found that a single data structure
inside Tomcat grew by 240MB in a single day. It is the
'recycledRequestsAndResponses' stack inside Http2Protocol.java. This
data structure was introduced in the 3 latest commits on
Http2Protocol.java:
https://github.com/apache/tomcat/
commits/9669f73517971e2b45280979a63b8153585cddc8/java/org/apache/
coyote/http2/Http2Protocol.java
It was introduced in Tomcat 10.1.27 (not released) and released as
part of Tomcat 10.1.28:
Align HTTP/2 with HTTP/1.1 and recycle the container internal
request and response processing objects by default. This behavior
can be controlled via the new discardRequestsAndResponses attribute
on the HTTP/2 upgrade protocol. (markt)
It only applies to HTTP/2 requests, and we get a steady stream of
those from other internal services. Requests from the outside world
are always HTTP 1.1. As you can see in the Memory Analyzer
screenshot, each request in the stack is rather large (360KB). There
are byte arrays of 65535 bytes that are only filled with 200 characters.
There is a limit to the stack which is set in this line:
recycledRequestsAndResponses.setLimit(http11Protocol.getMaxConnections());
In our case the maximum number of connections is 8192. So. the stack
could theoretically become as large as 8192 x 360KB (almost 3 Gigabyte).
I do see code to "recyle" (i.e. clear) the request before putting it
onto the stack:
requestAndResponse.recycle();
This clears a lot of fields but (probably intentionally) not some of
the larger fields.
Fortunately, we can disable this mechanism, and we have just deployed
a version of our app with 'discardRequestsAndResponses' set to true
by customizing the 'TomcatServletWebServerFactory'. After running
for two and a half hours, we can already see that our app now
consumes memory at a slower rate.
Are we doing something wrong? Is the maximum number of connections
unreasonably high? Is there some reason why our Request objects
become so large?
Kind regards, Arjan van IJzendoorn
[cid:21200d4c-68ac-46b8-8dc8-affc43f6475d]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
