RE: Increase in GC Pause time in tomcat http2 9.0.56
Any comments on this please . Thanks and Regards Arshiya Shariff -Original Message- From: Arshiya Shariff Sent: 30 August 2022 20:08 To: Tomcat Users List Subject: RE: Increase in GC Pause time in tomcat http2 9.0.56 Thanks Zdeněk for the response . 1)We have enabled GC logs and we see that mixed GC takes more than 1 sec and it occurs once in 2 hours approximately , whereas with tomcat 9.0.38 mixed GC occurs every 20 mins approximately with very small GC Pause, both the GC log comparisons i.e. with tomcat 9.0.56 and 9.0.38 were done with the same object allocation rate 2022-08-18T12:58:47.240-0400: 4956324.814: [GC pause (G1 Evacuation Pause) (mixed), 1.8309540 secs] [Parallel Time: 1778.2 ms, GC Workers: 6] [GC Worker Start (ms): Min: 4956324838.5, Avg: 4956324838.9, Max: 4956324839.2, Diff: 0.7] [Ext Root Scanning (ms): Min: 19.5, Avg: 19.9, Max: 20.9, Diff: 1.4, Sum: 119.6] [Update RS (ms): Min: 5.4, Avg: 5.4, Max: 5.5, Diff: 0.0, Sum: 32.6] [Processed Buffers: Min: 71, Avg: 109.3, Max: 128, Diff: 57, Sum: 656] [Scan RS (ms): Min: 1137.0, Avg: 1138.0, Max: 1138.6, Diff: 1.6, Sum: 6827.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [Object Copy (ms): Min: 613.2, Avg: 613.8, Max: 614.8, Diff: 1.6, Sum: 3682.5] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 1.8] [Termination Attempts: Min: 1, Avg: 893.3, Max: 1141, Diff: 1140, Sum: 5360] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5] [GC Worker Total (ms): Min: 1777.1, Avg: 1777.5, Max: 1777.9, Diff: 0.8, Sum: 10664.9] [GC Worker End (ms): Min: 4956326616.4, Avg: 4956326616.4, Max: 4956326616.5, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.8 ms] [Other: 51.0 ms] [Choose CSet: 23.1 ms] [Ref Proc: 2.4 ms] [Ref Enq: 0.2 ms] [Redirty Cards: 2.6 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 20.8 ms] [Eden: 2144.0M(2144.0M)->0.0B(2128.0M) Survivors: 104.0M->120.0M Heap: 6632.0M(22.0G)->3296.0M(22.0G)] [Times: user=8.61 sys=2.00, real=1.83 secs] 2)The 5gb of swap that is used is not completely by this process , we have few other processes running . 3)These parameters in production are fine-tuned for the process. Thanks and Regards Arshiya Shariff -Original Message- From: Zdeněk Henek Sent: 30 August 2022 19:43 To: Tomcat Users List Subject: Re: Increase in GC Pause time in tomcat http2 9.0.56 HI Arshiya, MaxGCPauseMillis=100 maybe this is too strict? Causing you run into full gc. Jdk8 G1GC implementation has single thread full gc and these take more time to finish? Just guessing as you have not written if the long pause gcs are young gen pauses or full gc pauses. Another option which could cause longer gc pauses could be usage of swap file / swap partition. Check and monitor how much memory is used and how the swap is used. Your system has 120gb of ram. That is a lot, but still it does use approx 18gb swap file! There is used 5gb of that swap. I think it may be better to remove that swap file completely or reduce swappiness param in your OS. There is assigned 22gb heap, G1GC in jdk8 uses a lot of memory for G1 GC metadata. It could be up to 6gb in your case. I would consider upgrade to jdk17 and plan upgrades to newer jdk frequently if you can. There are significant improvements in G1GC in new jdk versions, not only metadatas size. I would start troubleshooting removing all gc related params and let G1GC decide all the params during runtime. It is usually a good starting point to troubleshooting G1GC if you don't know what is wrong. Enable gc logs and use gc log analyser to see what is going on there. There is page with list of available GC analyzers here https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-3a2e7b00ac43e57a=1=b7c12870-06e4-4332-b0b8-6462d5cfa43d=http%3A%2F%2Ffasterj.com%2Ftools%2Fgcloganalysers.shtml Regards, Zdenek On Tue, Aug 30, 2022 at 3:31 PM Arshiya Shariff wrote: > We haven’t made any changes to the gc params when migrating tomcat : > > Please find the configurations: > > Java HotSpot(TM) 64-Bit Server VM (25.241-b07) for linux-amd64 JRE > (1.8.0_241-b07), built on Dec 11 2019 02:22:16 by "java_re" with gcc > 7.3.0 > Memory: 4k page, physical 119478416k(781944k free), swap > 18862076k(13762004k free) > -XX:ConcGCThreads=3 -XX:G1HeapRegionSize=8388608 > -XX:G1NewSizePercent=10 > -XX:G1ReservePercent=20 -XX:InitialHeapSize=23622320128 > -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=100 > -XX:MaxHeapSize=23622320128 -XX:MaxMetaspaceSize=268435456 > -XX:MaxNewSize=14168358912 -XX:MinHeapDeltaBytes=8388608 > -XX:ParallelGCThreads=6 -XX:+UnlockExperimentalVMOptions > -XX:+UseCompressedClassPoi
RE: Increase in GC Pause time in tomcat http2 9.0.56
Thanks Zdeněk for the response . 1)We have enabled GC logs and we see that mixed GC takes more than 1 sec and it occurs once in 2 hours approximately , whereas with tomcat 9.0.38 mixed GC occurs every 20 mins approximately with very small GC Pause, both the GC log comparisons i.e. with tomcat 9.0.56 and 9.0.38 were done with the same object allocation rate 2022-08-18T12:58:47.240-0400: 4956324.814: [GC pause (G1 Evacuation Pause) (mixed), 1.8309540 secs] [Parallel Time: 1778.2 ms, GC Workers: 6] [GC Worker Start (ms): Min: 4956324838.5, Avg: 4956324838.9, Max: 4956324839.2, Diff: 0.7] [Ext Root Scanning (ms): Min: 19.5, Avg: 19.9, Max: 20.9, Diff: 1.4, Sum: 119.6] [Update RS (ms): Min: 5.4, Avg: 5.4, Max: 5.5, Diff: 0.0, Sum: 32.6] [Processed Buffers: Min: 71, Avg: 109.3, Max: 128, Diff: 57, Sum: 656] [Scan RS (ms): Min: 1137.0, Avg: 1138.0, Max: 1138.6, Diff: 1.6, Sum: 6827.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [Object Copy (ms): Min: 613.2, Avg: 613.8, Max: 614.8, Diff: 1.6, Sum: 3682.5] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 1.8] [Termination Attempts: Min: 1, Avg: 893.3, Max: 1141, Diff: 1140, Sum: 5360] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 0.5] [GC Worker Total (ms): Min: 1777.1, Avg: 1777.5, Max: 1777.9, Diff: 0.8, Sum: 10664.9] [GC Worker End (ms): Min: 4956326616.4, Avg: 4956326616.4, Max: 4956326616.5, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.8 ms] [Other: 51.0 ms] [Choose CSet: 23.1 ms] [Ref Proc: 2.4 ms] [Ref Enq: 0.2 ms] [Redirty Cards: 2.6 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 20.8 ms] [Eden: 2144.0M(2144.0M)->0.0B(2128.0M) Survivors: 104.0M->120.0M Heap: 6632.0M(22.0G)->3296.0M(22.0G)] [Times: user=8.61 sys=2.00, real=1.83 secs] 2)The 5gb of swap that is used is not completely by this process , we have few other processes running . 3)These parameters in production are fine-tuned for the process. Thanks and Regards Arshiya Shariff -Original Message- From: Zdeněk Henek Sent: 30 August 2022 19:43 To: Tomcat Users List Subject: Re: Increase in GC Pause time in tomcat http2 9.0.56 HI Arshiya, MaxGCPauseMillis=100 maybe this is too strict? Causing you run into full gc. Jdk8 G1GC implementation has single thread full gc and these take more time to finish? Just guessing as you have not written if the long pause gcs are young gen pauses or full gc pauses. Another option which could cause longer gc pauses could be usage of swap file / swap partition. Check and monitor how much memory is used and how the swap is used. Your system has 120gb of ram. That is a lot, but still it does use approx 18gb swap file! There is used 5gb of that swap. I think it may be better to remove that swap file completely or reduce swappiness param in your OS. There is assigned 22gb heap, G1GC in jdk8 uses a lot of memory for G1 GC metadata. It could be up to 6gb in your case. I would consider upgrade to jdk17 and plan upgrades to newer jdk frequently if you can. There are significant improvements in G1GC in new jdk versions, not only metadatas size. I would start troubleshooting removing all gc related params and let G1GC decide all the params during runtime. It is usually a good starting point to troubleshooting G1GC if you don't know what is wrong. Enable gc logs and use gc log analyser to see what is going on there. There is page with list of available GC analyzers here https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-3a2e7b00ac43e57a=1=b7c12870-06e4-4332-b0b8-6462d5cfa43d=http%3A%2F%2Ffasterj.com%2Ftools%2Fgcloganalysers.shtml Regards, Zdenek On Tue, Aug 30, 2022 at 3:31 PM Arshiya Shariff wrote: > We haven’t made any changes to the gc params when migrating tomcat : > > Please find the configurations: > > Java HotSpot(TM) 64-Bit Server VM (25.241-b07) for linux-amd64 JRE > (1.8.0_241-b07), built on Dec 11 2019 02:22:16 by "java_re" with gcc > 7.3.0 > Memory: 4k page, physical 119478416k(781944k free), swap > 18862076k(13762004k free) > -XX:ConcGCThreads=3 -XX:G1HeapRegionSize=8388608 > -XX:G1NewSizePercent=10 > -XX:G1ReservePercent=20 -XX:InitialHeapSize=23622320128 > -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=100 > -XX:MaxHeapSize=23622320128 -XX:MaxMetaspaceSize=268435456 > -XX:MaxNewSize=14168358912 -XX:MinHeapDeltaBytes=8388608 > -XX:ParallelGCThreads=6 -XX:+UnlockExperimentalVMOptions > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > -XX:+UseGCLogFileRotation > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Rob Sargent > Sent: 30 August 2022 18:22 > To: users@
RE: Increase in GC Pause time in tomcat http2 9.0.56
We haven’t made any changes to the gc params when migrating tomcat : Please find the configurations: Java HotSpot(TM) 64-Bit Server VM (25.241-b07) for linux-amd64 JRE (1.8.0_241-b07), built on Dec 11 2019 02:22:16 by "java_re" with gcc 7.3.0 Memory: 4k page, physical 119478416k(781944k free), swap 18862076k(13762004k free) -XX:ConcGCThreads=3 -XX:G1HeapRegionSize=8388608 -XX:G1NewSizePercent=10 -XX:G1ReservePercent=20 -XX:InitialHeapSize=23622320128 -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=100 -XX:MaxHeapSize=23622320128 -XX:MaxMetaspaceSize=268435456 -XX:MaxNewSize=14168358912 -XX:MinHeapDeltaBytes=8388608 -XX:ParallelGCThreads=6 -XX:+UnlockExperimentalVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation Thanks and Regards Arshiya Shariff -Original Message- From: Rob Sargent Sent: 30 August 2022 18:22 To: users@tomcat.apache.org Subject: Re: Increase in GC Pause time in tomcat http2 9.0.56 On 8/30/22 06:11, Arshiya Shariff wrote: > Hi All , > We have migrated our application (which transports http/2 packets) from > embedded tomcat version 9.0.38 to 9.0.56 in production . > > We observe that , in 9.0.56 mixed GC occurs less frequently than in 9.0.38 , > but when it happens there are GC Pauses for more than 1 second , which is > causing the application to be unresponsive during that time and as a side > effect we are facing other issues like delay in WINDOW_UPDATE frames being > sent. > > Has anyone faced similar issue with increased GC Pause time ? > > Thanks and Regards > Arshiya Shariff > And no change to the JVM? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Increase in GC Pause time in tomcat http2 9.0.56
Hi All , We have migrated our application (which transports http/2 packets) from embedded tomcat version 9.0.38 to 9.0.56 in production . We observe that , in 9.0.56 mixed GC occurs less frequently than in 9.0.38 , but when it happens there are GC Pauses for more than 1 second , which is causing the application to be unresponsive during that time and as a side effect we are facing other issues like delay in WINDOW_UPDATE frames being sent. Has anyone faced similar issue with increased GC Pause time ? Thanks and Regards Arshiya Shariff
RE: HTTP2 : WINDOW_UPDATE not sent on stream level
Thank you for the response Mark . Yes, with the default window size the requests process fine . We were just attempting to reproduce the original issue that we are facing in production with version 9.0.38 where few connections go unresponsive for few seconds around the same time i.e. tomcat is able to receive the headers/partial data, but not responding back with window_update or sending ping frames and then after sometime tomcat sends PING frame and the connection processes further requests successfully . The GC looks fine at that time. Could there be any reasons that can cause this ? Please let us know. Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, January 5, 2022 2:36 PM To: users@tomcat.apache.org Subject: Re: HTTP2 : WINDOW_UPDATE not sent on stream level On 05/01/2022 06:14, Arshiya Shariff wrote: > Hi Team, > On sending 3 requests of around 3KB size , we see that only the first > request has processed fine. The other 2 requests are waiting for more Data as > tomcat has not responded with WINDOW_UPDATE on stream level . Please help us > understand this behavior . > > Embedded tomcat version : 9.0.56 > The initial window size is set to 500, all other params are the tomcat > defaults . > HTTP2 port : 1080 > Client : JMeter > > I have placed the reproducer , debug logs ,PCAP and JMX where we have tested > with version 9.0.56 here : > https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444 > 731-d40684d89713d0eb=1=8a27bb36-1eab-4d6a-bee6-03760f68d928= > https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1u4Sc_7oSB9tdJN8ls9xTMrRqP > 8IFoACM%2Fview%3Fusp%3Dsharing > > The same JMX works fine with the overhead values changed to zero. Which indicates the requests you are sending are triggering the excessive overhead protection code. > Will we have any side effects on changing the overhead parameters to zero? For Tomcat, generally not since it wasn't vulnerable to the various HTTP/2 DoS attacks that were popular a few years ago. The overhead protection was added primarily as a way to detect and reject clients exhibiting abusive behaviour. Looking at your source code you have set the HTTP/2 initial window size to 500. That is far too low. Since it is lower than the default overhead threshold for data (1024) it means every single DATA frame is going to be classed as abusive. No wonder the connection gets closed down so quickly. If you set the default window size to something sensible - or just leave it as the default - do you still see the issue? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP2 : WINDOW_UPDATE not sent on stream level
Hi Team, On sending 3 requests of around 3KB size , we see that only the first request has processed fine. The other 2 requests are waiting for more Data as tomcat has not responded with WINDOW_UPDATE on stream level . Please help us understand this behavior . Embedded tomcat version : 9.0.56 The initial window size is set to 500, all other params are the tomcat defaults . HTTP2 port : 1080 Client : JMeter I have placed the reproducer , debug logs ,PCAP and JMX where we have tested with version 9.0.56 here : https://drive.google.com/file/d/1u4Sc_7oSB9tdJN8ls9xTMrRqP8IFoACM/view?usp=sharing The same JMX works fine with the overhead values changed to zero. Will we have any side effects on changing the overhead parameters to zero? Thanks in Advance! Regards Arshiya Shariff
RE: Embedded Tomcat 9.0.43 : WINDOW_UPDATE not sent when receiving http2 requests over unknown url
Thank you Mark . Please keep us posted regarding the fix . -Original Message- From: Mark Thomas Sent: Wednesday, March 10, 2021 6:28 PM To: users@tomcat.apache.org Subject: Re: Embedded Tomcat 9.0.43 : WINDOW_UPDATE not sent when receiving http2 requests over unknown url On 10/03/2021 05:26, Arshiya Shariff wrote: > Hi All, > We are using embedded tomcat version 9.0.43 in our application to transport > http/2 packets between 2 systems (h2c connection). All parameters used are > the tomcat defaults. > > We are facing the below issue : > >1. Tomcat is not sending WINDOW_UPDATE when a request(payload > 65K) is > landed on an Unknown URL(which is not deployed as a servlet and not having a > servlet mapping). The same is working when request landed on an known > URL(which is deployed as a servlet and has an servlet mapping). >2. Tomcat is listening on 1080 port, client is sending packet size of > > 65KB, but here tomcat receives only 65KB and not receiving full DATA, later > tomcat sends RST_STREAM, further no WINDOW_UPDATE, after that client is not > able to send DATA frames as there is no WINDOW_UPDATE from tomcat. > > > Kindly let us know whether it is a bug or not. Certainly looks like a bug. I'll investigate now... Mark > > Working case : > Client is sending to an correct URL of size/payload > 65KB. Tomcat is sending > WINDOW_UPDATE. 200OK from application, which is working properly. Later > client is able to send requests along with DATA also, and tomcat is receiving > those > > I have placed the PCAPs for the Working(correct url) and Not Working Case > (unknown url) in the below path: > > https://protect2.fireeye.com/v1/url?k=e9471c24-b6dc2509-e9475cbf-8692dc8284cb-4463215f54d09c7e=1=77145d97-7041-4d7c-bf52-6bccde1b4a62=https%3A%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1IN4XVQe2cvGIIQbHmAu-XLeWhjb-rykr%3Fusp%3Dsharing > > > Thanks and Regards > Arshiya Shariff > > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Embedded Tomcat 9.0.43 : WINDOW_UPDATE not sent when receiving http2 requests over unknown url
Hi All, We are using embedded tomcat version 9.0.43 in our application to transport http/2 packets between 2 systems (h2c connection). All parameters used are the tomcat defaults. We are facing the below issue : 1. Tomcat is not sending WINDOW_UPDATE when a request(payload > 65K) is landed on an Unknown URL(which is not deployed as a servlet and not having a servlet mapping). The same is working when request landed on an known URL(which is deployed as a servlet and has an servlet mapping). 2. Tomcat is listening on 1080 port, client is sending packet size of > 65KB, but here tomcat receives only 65KB and not receiving full DATA, later tomcat sends RST_STREAM, further no WINDOW_UPDATE, after that client is not able to send DATA frames as there is no WINDOW_UPDATE from tomcat. Kindly let us know whether it is a bug or not. Working case : Client is sending to an correct URL of size/payload > 65KB. Tomcat is sending WINDOW_UPDATE. 200OK from application, which is working properly. Later client is able to send requests along with DATA also, and tomcat is receiving those I have placed the PCAPs for the Working(correct url) and Not Working Case (unknown url) in the below path: https://drive.google.com/drive/folders/1IN4XVQe2cvGIIQbHmAu-XLeWhjb-rykr?usp=sharing Thanks and Regards Arshiya Shariff
RE: FW: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi all,
Thanks for the update Christopher.
I tried modifying our application as below i.e by adding a read Listener and
processing the requests after all the data is read (onDataAvailable()) .
But I can still see the StackOverFlow Error printed .
AsyncContext asyncContext = req.startAsync( req, resp );
ServletInputStream input = req.getInputStream();
ReadListenerImpl listener = new ReadListenerImpl( input,
asyncContext, req, resp, map, myExecutor, this );
input.setReadListener( listener );
. . .
public class ReadListenerImpl implements ReadListener
{
.
.
.
@Override
public void onDataAvailable() throws IOException
{
int len = -1;
byte b[] = new byte[4 * 1024];
while( input.isReady() &&
!input.isFinished() )
{
len = input.read( b );
if( len > 0 )
{
String data = new
String( b, 0, len );
sb.append( data );
}
}
}
@Override
public void onError(Throwable throwable)
{
asyncContext.complete();
throwable.printStackTrace();
}
@Override
public void onAllDataRead() throws IOException
{
String jsonData = sb.toString();
.
.
.
}
}
Unfortunately I am still not able to reproduce this exception with a test code
, I am trying .
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Christopher Schultz
Sent: Wednesday, October 21, 2020 7:53 PM
To: users@tomcat.apache.org
Subject: Re: FW: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
Arshiya,
On 10/21/20 00:34, Arshiya Shariff wrote:
> Hi,
>
> Christopher, Please find the answer in-line:
> How... exactly?
> private String getRequestBody(HttpServletRequest request) throws IOException
> {
> StringBuilder sb = new StringBuilder();
> BufferedReader reader = request.getReader();
> try
> {
> String line;
> while( ( line = reader.readLine() ) != null )
> {
> sb.append( line ).append( '\n' );
Note that this may modify the incoming request. Are you sure you want to return
a value which does not match the exact POST body?
> }
> }
> finally
> {
> reader.close();
> }
> return sb.toString();
> }
>
Is that method run from within the asynchronous context or before you begin
async processing? I'm not an expert at servlet-async, but I think you should be
reading the request entirely before entering the async context. Reading the
request from async may cause problems.
Instead of using blocking reads of the request bbody in asynchronous mode, you
should do this:
request.getInputStream().setReadListener(new ReadListener() {
public void onoAllDataRead() { ... }
public void onDataAvailable() { ... }
public void onError(Throwable t) { ... } });
> I am trying to reproduce the StackOverflowError with a sample
> application , Once it is reproduced I will share it across.
See https://www.slideshare.net/SimoneBordet/servlet-31-async-io
Specifically slides 43-53.
-chris
> -Original Message-
> From: Christopher Schultz
> Sent: Thursday, October 15, 2020 12:01 AM
> To: users@tomcat.apache.org
> Subject: Re: FW: HTTP2: memory filled up fast on increasing the
> connections to 1000/2000 (Embedded tomcat 9.0.38)
>
> Arshiya,
>
> On 10/14/20 01:23, Arshiya Shariff wrote:
>> Please find the answers in-line Mark.
>>
RE: FW: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi,
Christopher, Please find the answer in-line:
How... exactly?
private String getRequestBody(HttpServletRequest request) throws IOException
{
StringBuilder sb = new StringBuilder();
BufferedReader reader = request.getReader();
try
{
String line;
while( ( line = reader.readLine() ) != null )
{
sb.append( line ).append( '\n' );
}
}
finally
{
reader.close();
}
return sb.toString();
}
I am trying to reproduce the StackOverflowError with a sample application ,
Once it is reproduced I will share it across.
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Christopher Schultz
Sent: Thursday, October 15, 2020 12:01 AM
To: users@tomcat.apache.org
Subject: Re: FW: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
Arshiya,
On 10/14/20 01:23, Arshiya Shariff wrote:
> Please find the answers in-line Mark.
>
> Http2 requests with message payload of 34KB are pumped from JMeter at
> 20 TPS with 700 connections to an application with Embedded tomcat
> - 9.0.39 (max-Threads : 200, all other values are the tomcat
> defaults)
>
>> What does that URL do with the POSTed content? Ignore it? Read it
>> from an InputStream? Read it via getParameter()?
>
> The posted content is read via BufferedReader reader =
> request.getReader() and processed asynchronously
How... exactly?
> Is JMeter run on the same machine as Tomcat?
> JMeter is run from a different machine.
>
> Do you use the JMeter GUI or the command line?
> Launched via Command line (JMeter heap increased to 10 GB )
>
> What are the specs of the server(s) being used?
> The server is a VM with 12 CPUs and 120 GB RAM
>
> Please let us know if you require more details.
This would probabyl be easier if you'd just provide a test-case: a sample
(simple!) web application which reproduces what you are reporting.
-chris
> -Original Message-
> From: Mark Thomas
> Sent: Monday, October 12, 2020 7:28 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the
> connections to 1000/2000 (Embedded tomcat 9.0.38)
>
> On 12/10/2020 08:02, Arshiya Shariff wrote:
>> Hi Mark ,
>>
>> The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat
>> is 200.
>>
>> Please find the case:
>> Client:JMeter 5.2.1 (With http2 plugin)
>> TPS: around 20
>> No of users from JMeter : 700
>> Message payload size: 6 KB to 34 KB
>> Loop: Infinite
>> We let the loop run infinitely and see the java.lang.StackOverflowError
>> trace printed multiple times in the log within few minutes of starting the
>> test.
>
> POSTing to what URL?
>
> What does that URL do with the POSTed content? Ignore it? Read it from an
> InputStream? Read it via getParameter()?
>
> Is JMeter run on the same machine as Tomcat?
>
> Do you use the JMeter GUI or the command line?
>
> What are the specs of the server(s) being used?
>
> You need to provide the exact steps to recreate this issue on a clean install
> of Tomcat 9.0.39 as provided by the ASF.
>
> Mark
>
>
>> Please help us with this . What is the impact of StackOverflowError ?
>>
>> Thanks and Regards
>> Arshiya Shariff
>>
>> -----Original Message-
>> From: Mark Thomas
>> Sent: Friday, October 9, 2020 5:31 PM
>> To: users@tomcat.apache.org
>> Subject: Re: HTTP2: memory filled up fast on increasing the
>> connections to 1000/2000 (Embedded tomcat 9.0.38)
>>
>> On 09/10/2020 12:32, Arshiya Shariff wrote:
>>> Hi,
>>>
>>> Mark , with the test runs that I performed over clean 9.0.x branch I was
>>> not able to reproduce this.
>>
>> Good. But I'd really like to understand why...
>>
>>> But with 9.0.38 and the jars built from 9.0.x with hash:
>>> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users
>>> (connections) and on sending 1000 Requests per second (or even lesser) ,
>>> payload of 16K from JMeter I can see that this Exception occurs within few
>>> minutes of starting the test . The maxThreads configured in tomcat is 200 .
>>>
>>> How often do you see these errors in your test run?
>>> Randomly, at times 2 or 3 such traces.
>>
>> OK. Definitely a timing issue then.
>>
>>> Do you have the other end of th
FW: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi , Please find the answers in-line Mark. Http2 requests with message payload of 34KB are pumped from JMeter at 20 TPS with 700 connections to an application with Embedded tomcat - 9.0.39 (max-Threads : 200, all other values are the tomcat defaults) What does that URL do with the POSTed content? Ignore it? Read it from an InputStream? Read it via getParameter()? The posted content is read via BufferedReader reader = request.getReader() and processed asynchronously. Is JMeter run on the same machine as Tomcat? JMeter is run from a different machine. Do you use the JMeter GUI or the command line? Launched via Command line (JMeter heap increased to 10 GB ) What are the specs of the server(s) being used? The server is a VM with 12 CPUs and 120 GB RAM Please let us know if you require more details. Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Monday, October 12, 2020 7:28 PM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 12/10/2020 08:02, Arshiya Shariff wrote: > Hi Mark , > > The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is > 200. > > Please find the case: > Client:JMeter 5.2.1 (With http2 plugin) > TPS: around 20 > No of users from JMeter : 700 > Message payload size: 6 KB to 34 KB > Loop: Infinite > We let the loop run infinitely and see the java.lang.StackOverflowError trace > printed multiple times in the log within few minutes of starting the test. POSTing to what URL? What does that URL do with the POSTed content? Ignore it? Read it from an InputStream? Read it via getParameter()? Is JMeter run on the same machine as Tomcat? Do you use the JMeter GUI or the command line? What are the specs of the server(s) being used? You need to provide the exact steps to recreate this issue on a clean install of Tomcat 9.0.39 as provided by the ASF. Mark > Please help us with this . What is the impact of StackOverflowError ? > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Mark Thomas > Sent: Friday, October 9, 2020 5:31 PM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on increasing the > connections to 1000/2000 (Embedded tomcat 9.0.38) > > On 09/10/2020 12:32, Arshiya Shariff wrote: >> Hi, >> >> Mark , with the test runs that I performed over clean 9.0.x branch I was not >> able to reproduce this. > > Good. But I'd really like to understand why... > >> But with 9.0.38 and the jars built from 9.0.x with hash: >> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users >> (connections) and on sending 1000 Requests per second (or even lesser) , >> payload of 16K from JMeter I can see that this Exception occurs within few >> minutes of starting the test . The maxThreads configured in tomcat is 200 . >> >> How often do you see these errors in your test run? >> Randomly, at times 2 or 3 such traces. > > OK. Definitely a timing issue then. > >> Do you have the other end of that stack trace? >> It is only the two lines that is recursively printed till the end about >> ~500 times in one trace : >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHand >> l >> er.completed(SocketWrapperBase.java:1100) > > Doesn't tell me much unfortunately. > >> I see the trace starting with : >> Exception in thread "http-nio-x.y.z-1090-exec-107" >> java.lang.StackOverflowError >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) >> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHand >> l >> er.completed(SocketWrapperBase.java:1100) >> >> (OR) >> >> Exception in thread "http-nio-x.y.z-1090-exec-87" >> java.lang.StackOverflowError >> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) >> at sun.nio.ch.IOUtil.read(IOUtil.java:240) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) >> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi Mark , The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat is 200. Please find the case: Client:JMeter 5.2.1 (With http2 plugin) TPS: around 20 No of users from JMeter : 700 Message payload size: 6 KB to 34 KB Loop: Infinite We let the loop run infinitely and see the java.lang.StackOverflowError trace printed multiple times in the log within few minutes of starting the test. Please help us with this . What is the impact of StackOverflowError ? Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Friday, October 9, 2020 5:31 PM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 09/10/2020 12:32, Arshiya Shariff wrote: > Hi, > > Mark , with the test runs that I performed over clean 9.0.x branch I was not > able to reproduce this. Good. But I'd really like to understand why... > But with 9.0.38 and the jars built from 9.0.x with hash: > c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users > (connections) and on sending 1000 Requests per second (or even lesser) , > payload of 16K from JMeter I can see that this Exception occurs within few > minutes of starting the test . The maxThreads configured in tomcat is 200 . > > How often do you see these errors in your test run? > Randomly, at times 2 or 3 such traces. OK. Definitely a timing issue then. > Do you have the other end of that stack trace? > It is only the two lines that is recursively printed till the end about ~500 > times in one trace : > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) Doesn't tell me much unfortunately. > I see the trace starting with : > Exception in thread "http-nio-x.y.z-1090-exec-107" > java.lang.StackOverflowError > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) > > (OR) > > Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError > at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > . > . > . > . > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandl > er.completed(SocketWrapperBase.java:1100) > > Is there anything that was fixed around this in latest 9.0.x branch ? Not obviously. I've reviewed every commit since c8ec2d4c. There is nothing that directly works with the I/O. There is 1e97ab2 which fixes a relatively recent regression in the HTTP/2 code. I guess it is possible (but it seems a bit of a stretch) that that bug is triggering an issue in JMeter which in turn is sending invalid HTTP/2 packets. I think at this point, given the relatively small number of commits between c8ec2d4c and HEAD, the most useful thing you could do is run a binary search to find out at which commit the issue is fixed. If we know which commit to look at that should help track down the root cause. Mark > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Mark Thomas > Sent: Monday, October 5, 2020 9:52 PM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi, Mark , with the test runs that I performed over clean 9.0.x branch I was not able to reproduce this. But with 9.0.38 and the jars built from 9.0.x with hash: c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users (connections) and on sending 1000 Requests per second (or even lesser) , payload of 16K from JMeter I can see that this Exception occurs within few minutes of starting the test . The maxThreads configured in tomcat is 200 . How often do you see these errors in your test run? Randomly, at times 2 or 3 such traces. Do you have the other end of that stack trace? It is only the two lines that is recursively printed till the end about ~500 times in one trace : at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) I see the trace starting with : Exception in thread "http-nio-x.y.z-1090-exec-107" java.lang.StackOverflowError at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446) at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) (OR) Exception in thread "http-nio-x.y.z-1090-exec-87" java.lang.StackOverflowError at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96) at sun.nio.ch.IOUtil.read(IOUtil.java:240) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) . . . . at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) Is there anything that was fixed around this in latest 9.0.x branch ? Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Monday, October 5, 2020 9:52 PM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 05/10/2020 10:56, Arshiya Shariff wrote: > Hi All, > > Thank you so much Mark . > We tested the jars built from latest 9.0.x with 2000 / 5000 users > (connections) from JMeter , We see a very good improvement with the > heap usage Good news. As is the fact that the other errors have been cleared up. > But I see this exception printed multiple times , I am not sure why this > occurs : > Exception in thread "http-nio-x.y.z-1234-exec-213" > java.lang.StackOverflowError > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationSt > ate.run(NioEndpoint.java:1511) That looks like an infinite loop reading an incoming frame. New frames are read using a 9 byte buffer for the header and a 16k buffer for the payload (since Tomcat sets this as the max frame size)
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi All,
Thank you so much Mark .
We tested the jars built from latest 9.0.x with 2000 / 5000 users
(connections) from JMeter , We see a very good improvement with the heap usage .
But I see this exception printed multiple times , I am not sure why this occurs
:
Exception in thread "http-nio-x.y.z-1234-exec-213" java.lang.StackOverflowError
at sun.nio.ch.IOUtil.read(IOUtil.java:240)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
Any help with this please.
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Mark Thomas
Sent: Thursday, October 1, 2020 2:59 PM
To: users@tomcat.apache.org
Subject: Re: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 18:47, Martin Grigorov wrote:
> On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas wrote:
>> On 30/09/2020 16:17, Mark Thomas wrote:
>>> That is helpful. Looks like you have found a way to reproduce the
>>> buffer issues reported in
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>>
>> Can you share the command you used to trigger those errors please.
>>
>
> The Vegeta command I used is:
>
> jq -ncM '{"method": "POST", "url":
> "https://localhost:8080/testbed/plaintext;,
> "body":"payload=Some
> sdgggwwsdgssfshffheeepayload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta
> encode > /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The app is at
> https://protect2.fireeye.com/v1/url?k=c248c6cb-9ce86806-c2488650-866132fe445e-bcef5199a0b9f57e=1=5f78363c-a75a-4bc1-b339-a919d2804f90=https%3A%2F%2Fgithub.com%2Fmartin-g%2Fhttp2-server-perf-tests%2Ftree%2Fmaster%2Fjava%2Ftomcat.
> Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
Definitely timing related as I am unable to reproduce the problem with that
command or some variations.
However, I think I have managed to track down the root cause. The good news is
that the BufferOverflowException is largely harmless. It is a side-effect of
the connection being closed due to an error. My guess is that the error was a
combination of vegeta sending an unexpected reset frame and Tomcat maintaining
state for a very small number of streams in some circumstances.
If you could retest with the latest 9.0.x that would be very helpful.
The memory usage, stream state maintenance and this BufferOverflowException
should all be fixed.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Thank you so much Mark .
We will test and keep you posted .
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Mark Thomas
Sent: Thursday, October 1, 2020 2:59 PM
To: users@tomcat.apache.org
Subject: Re: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 18:47, Martin Grigorov wrote:
> On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas wrote:
>> On 30/09/2020 16:17, Mark Thomas wrote:
>>> That is helpful. Looks like you have found a way to reproduce the
>>> buffer issues reported in
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>>
>> Can you share the command you used to trigger those errors please.
>>
>
> The Vegeta command I used is:
>
> jq -ncM '{"method": "POST", "url":
> "https://localhost:8080/testbed/plaintext;,
> "body":"payload=Some
> sdgggwwsdgssfshffheeepayload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta
> encode > /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The app is at
> https://protect2.fireeye.com/v1/url?k=c248c6cb-9ce86806-c2488650-866132fe445e-bcef5199a0b9f57e=1=5f78363c-a75a-4bc1-b339-a919d2804f90=https%3A%2F%2Fgithub.com%2Fmartin-g%2Fhttp2-server-perf-tests%2Ftree%2Fmaster%2Fjava%2Ftomcat.
> Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
Definitely timing related as I am unable to reproduce the problem with that
command or some variations.
However, I think I have managed to track down the root cause. The good news is
that the BufferOverflowException is largely harmless. It is a side-effect of
the connection being closed due to an error. My guess is that the error was a
combination of vegeta sending an unexpected reset frame and Tomcat maintaining
state for a very small number of streams in some circumstances.
If you could retest with the latest 9.0.x that would be very helpful.
The memory usage, stream state maintenance and this BufferOverflowException
should all be fixed.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Thank you for the response Mark , >> Are you able to test with a custom Tomcat build and/or build Tomcat 9 from >> source for testing? Yes Mark , we will be able to test with the jars built from Tomcat 9 source for testing . Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Tuesday, September 29, 2020 12:25 AM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 28/09/2020 17:58, Arshiya Shariff wrote: > Hi All, > With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and > by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat > application we did not observe any memory issue , but on sending 1000 http2 > requests/sec with 2000 or 1000 users from JMeter , the application's heap > space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory > clears and comes down to 4GB (expected) . > > Embedded tomcat Version:9.0.38 > Max Threads : 200 > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? You haven't provided any evidence that Tomcat isn't able to process "many" connections. > Why is the memory filled when the connections are increased, are there any > parameters to tune connections ? It looks like users == HTTP/2 Connection. Connections are required to maintain state for closed streams for both prioritisation and for error handling. More connections == more state == more memory. Given the number of connections increased by a factor of between 12.5 and 25, that the memory usage only increased by a factor of 5 looks to be a positive result rather than an issue. There are significant improvements to memory usage in this area in Tomcat 10.0.x that will get back-ported to 9.0.x but more testing is required. Are you able to test with a custom Tomcat build and/or build Tomcat 9 from source for testing? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi Martin , Thank you for the response. With a payload of 200 bytes we were able to send 20K requests/sec with 200 users from Jmeter without any memory issue . On increasing the payload to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests per second , the heap of 20GB got filled in 2 minutes . With 200 users the memory is cleared in the G1 mixed GC itself , but with 1000 users the memory is not cleared in the mixed GC , it takes full GCs of 7 to 10 seconds to clear the memory. These cases were executed with maxThreads 200 in tomcat , so we tried increasing the maxThreads from 200 to 1000, but still GC was struggling . When we tried with 10 instances of JMeter , each with 100 users , where each instance was started with a delay of 1 minute we were able to see 1000 connections created in tomcat without any memory issues. But when 1000 users are created using single instance of JMeter in 20 seconds , tomcat's memory is filling fast- 20GB in 2 minutes. We suspect that the burst of connections being opened has a problem . Please help us with the same . On analyzing the heap dump we see org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. Thanks and Regards Arshiya Shariff -Original Message- From: Martin Grigorov Sent: Monday, September 28, 2020 11:44 PM To: Tomcat Users List Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) Hi Arshiya, On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff wrote: > Hi All, > With 200 threads(users) , ramp up duration of 2 seconds , loop count > 80 and by sending 1000 http2 requests/sec from JMeter Client to an > embedded tomcat application we did not observe any memory issue , but > on sending > 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the > application's heap space of 20 GB is occupied in 2 minutes and after 2 > full GCs the memory clears and comes down to 4GB (expected) . > I am not sure whether you follow the other discussions at users@. In another email thread we discuss load testing Tomcat HTTP2 and we are able to make around 12K reqs/s with another load testing tool - https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f=1=6a9c569d-7da1-4394-a9ac-bf72724992fa=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta For me JMeter itself failed with OOM when increasing the number of the virtual users above 2K. There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. Some of the changes are not yet downported to 9.0.x. We still test them, trying to avoid introducing regressions in 9.0.x. > > Embedded tomcat Version:9.0.38 > Max Threads : 200 > The number of threads should be less if you do only CPU calculations without IO/network. If your app blocks on IO/network calls then you need more spare threads. With more threads there will be more context switches and less throughput. That's why there is no one golden rule that applies to all applications. 200 is a good default that works for most of the applications. But you need to test with different values to see which one gives the best performance for your scenaria. > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=. Once you have the .hprof file you can examine it with Eclipse Memory Analyzer tool and see what is leaking. I will try to reproduce this issue tomorrow with Vegeta. > Why is the memory filled when the connections are increased, are there > any parameters to tune connections ? > Please let us know. > > Thanks and Regards > Arshiya Shariff >
HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi All, With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat application we did not observe any memory issue , but on sending 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the application's heap space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory clears and comes down to 4GB (expected) . Embedded tomcat Version:9.0.38 Max Threads : 200 All other properties are the tomcat defaults. Why is tomcat not able to process many connections ? Why is the memory filled when the connections are increased, are there any parameters to tune connections ? Please let us know. Thanks and Regards Arshiya Shariff
RE: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR
Thanks for the response Martin. Below is the only exception we are getting in the logs while sending response to a closed stream , few times with MUST_COMPLETE instead of MUST_ERROR : Exception occurred while sending response. ExceptionMessage:Calling [asyncComplete()] is not valid for a request with Async state [MUST_ERROR] Thanks and Regards Arshiya Shariff -Original Message- From: Martin Grigorov Sent: Friday, September 25, 2020 4:37 PM To: Tomcat Users List Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR On Fri, Sep 25, 2020 at 1:18 PM Arshiya Shariff wrote: > Thanks Martin . > > Our expectation Is that , on receiving a RST_STREAM with CANCEL or > NO_ERROR from the client after 1.5 secs for a particular stream , we > don’t want the connection to be closed with a GOAWAY:NO_ERROR (while > trying to send response after 2secs asynchronously over a stream that > is closed) as the other streams during the same time are processing data fine > . > > Additional info for GOAWAY: Connection [], Stream[],An error occurred > during processing that was fatal to the connection . > I guess some other error happened and that led to the GOAWAY. But I cannot be sure. Any errors in the logs ? > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Martin Grigorov > Sent: Tuesday, September 22, 2020 7:31 PM > To: Tomcat Users List > Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a > stream where the client has already sent RST_STREAM:NO ERROR > > Hi, > > On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff < > arshiya.shar...@ericsson.com.invalid> wrote: > > > Thank you so much Martin for the response. > > Yes , 9.0.38 testing is on going . > > > > As we don’t get this clear with the RFC , please help us with the > > below two cases : > > * If a client sends RST_STREAM with NO_ERROR, then while sending > async > > response is it expected behavior to close connection with GOAWAY ? > > * If a client sends RST_STREAM with CANCEL , then while sending > > async response will tomcat send RST_STREAM or GOAWAY , from http2 > > protocol perspective ? > > > > As per https://tools.ietf.org/html/rfc7540#section-6.4 and > https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R" > are processed then the stream is moved to CLOSED state. > GOAWAY should be sent to the client only if the connection will be > closed ( https://tools.ietf.org/html/rfc7540#section-6.8). The server > should close the connection only if some serious problem has happened, > e.g. an IOException. > > Tell us more about your use case. What do you want to do when 1.5secs > pass ? What do you expect to happen ? > > > > > > Thanks and Regards > > Arshiya Sharif > > > > -Original Message- > > From: Martin Grigorov > > Sent: Tuesday, September 22, 2020 1:18 AM > > To: Tomcat Users List > > Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over > > a stream where the client has already sent RST_STREAM:NO ERROR > > > > Hi, > > > > On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff < > > arshiya.shar...@ericsson.com.invalid> wrote: > > > > > Hi All, > > > > > > The client has configured a response timeout of 1.5 seconds. In a > > > case when our application tries to respond over a http2 stream > > > asynchronously after 2 seconds where the client has already sent > > > RST_STREAM with NO ERROR in 1.5 seconds > > > > > > Why does the client send NO_ERROR to the server ? I think it should > > send a CANCEL instead. > > https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for > > "Graceful shutdown of the server". > > > > > > > (due to no response) , then tomcat sends GOAWAY and closes the > > > HTTP2 connection . Is this behavior of GOAWAY and connection > > > closure > expected ? > > > We have planned to upgrade to Embedded tomcat version 9.0.38 . > > > These are the behaviors we see in production with version 9.0.22 , > > > so we need some help with analyzing / validating the existing > > > behavior before > > the upgrade . > > > Please let us know. > > > > > > > Friendly advice: > > Please setup 9.0.38 locally and test on it. > > 9.0.22 is way too old. It is up to you to use it for your production > > but for reporting bugs it is recommended to use the latest available > version. > > I, pers
RE: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR
Thanks Martin . Our expectation Is that , on receiving a RST_STREAM with CANCEL or NO_ERROR from the client after 1.5 secs for a particular stream , we don’t want the connection to be closed with a GOAWAY:NO_ERROR (while trying to send response after 2secs asynchronously over a stream that is closed) as the other streams during the same time are processing data fine . Additional info for GOAWAY: Connection [], Stream[],An error occurred during processing that was fatal to the connection . Thanks and Regards Arshiya Shariff -Original Message- From: Martin Grigorov Sent: Tuesday, September 22, 2020 7:31 PM To: Tomcat Users List Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR Hi, On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff wrote: > Thank you so much Martin for the response. > Yes , 9.0.38 testing is on going . > > As we don’t get this clear with the RFC , please help us with the > below two cases : * If a client sends RST_STREAM with NO_ERROR, then while sending async > response is it expected behavior to close connection with GOAWAY ? > * If a client sends RST_STREAM with CANCEL , then while sending > async response will tomcat send RST_STREAM or GOAWAY , from http2 > protocol perspective ? > As per https://tools.ietf.org/html/rfc7540#section-6.4 and https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R" are processed then the stream is moved to CLOSED state. GOAWAY should be sent to the client only if the connection will be closed ( https://tools.ietf.org/html/rfc7540#section-6.8). The server should close the connection only if some serious problem has happened, e.g. an IOException. Tell us more about your use case. What do you want to do when 1.5secs pass ? What do you expect to happen ? > > Thanks and Regards > Arshiya Sharif > > -Original Message- > From: Martin Grigorov > Sent: Tuesday, September 22, 2020 1:18 AM > To: Tomcat Users List > Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a > stream where the client has already sent RST_STREAM:NO ERROR > > Hi, > > On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff < > arshiya.shar...@ericsson.com.invalid> wrote: > > > Hi All, > > > > The client has configured a response timeout of 1.5 seconds. In a > > case when our application tries to respond over a http2 stream > > asynchronously after 2 seconds where the client has already sent > > RST_STREAM with NO ERROR in 1.5 seconds > > > Why does the client send NO_ERROR to the server ? I think it should > send a CANCEL instead. > https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for > "Graceful shutdown of the server". > > > > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2 > > connection . Is this behavior of GOAWAY and connection closure expected ? > > We have planned to upgrade to Embedded tomcat version 9.0.38 . These > > are the behaviors we see in production with version 9.0.22 , so we > > need some help with analyzing / validating the existing behavior > > before > the upgrade . > > Please let us know. > > > > Friendly advice: > Please setup 9.0.38 locally and test on it. > 9.0.22 is way too old. It is up to you to use it for your production > but for reporting bugs it is recommended to use the latest available version. > I, personally, prefer to spend my spare time with my family and > friends than to debug old versions just because the user doesn't > bother to test on a newer version. > > > > > > Thanks and Regards > > Arshiya Shariff > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR
Thank you so much Martin for the response. Yes , 9.0.38 testing is on going . As we don’t get this clear with the RFC , please help us with the below two cases : * If a client sends RST_STREAM with NO_ERROR, then while sending async response is it expected behavior to close connection with GOAWAY ? * If a client sends RST_STREAM with CANCEL , then while sending async response will tomcat send RST_STREAM or GOAWAY , from http2 protocol perspective ? Thanks and Regards Arshiya Sharif -Original Message- From: Martin Grigorov Sent: Tuesday, September 22, 2020 1:18 AM To: Tomcat Users List Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR Hi, On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff wrote: > Hi All, > > The client has configured a response timeout of 1.5 seconds. In a case > when our application tries to respond over a http2 stream > asynchronously after 2 seconds where the client has already sent > RST_STREAM with NO ERROR in 1.5 seconds Why does the client send NO_ERROR to the server ? I think it should send a CANCEL instead. https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for "Graceful shutdown of the server". > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2 > connection . Is this behavior of GOAWAY and connection closure expected ? > We have planned to upgrade to Embedded tomcat version 9.0.38 . These > are the behaviors we see in production with version 9.0.22 , so we > need some help with analyzing / validating the existing behavior before the > upgrade . > Please let us know. > Friendly advice: Please setup 9.0.38 locally and test on it. 9.0.22 is way too old. It is up to you to use it for your production but for reporting bugs it is recommended to use the latest available version. I, personally, prefer to spend my spare time with my family and friends than to debug old versions just because the user doesn't bother to test on a newer version. > > Thanks and Regards > Arshiya Shariff > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR
Hi All, The client has configured a response timeout of 1.5 seconds. In a case when our application tries to respond over a http2 stream asynchronously after 2 seconds where the client has already sent RST_STREAM with NO ERROR in 1.5 seconds (due to no response) , then tomcat sends GOAWAY and closes the HTTP2 connection . Is this behavior of GOAWAY and connection closure expected ? We have planned to upgrade to Embedded tomcat version 9.0.38 . These are the behaviors we see in production with version 9.0.22 , so we need some help with analyzing / validating the existing behavior before the upgrade . Please let us know. Thanks and Regards Arshiya Shariff
RE: HTTP2: Connections abruptly closed by sending GOAWAY
Hi All , Any update on this please . Thanks and Regards Arshiya Shariff -Original Message- From: Arshiya Shariff Sent: Wednesday, September 9, 2020 6:49 PM To: Tomcat Users List Subject: RE: HTTP2: Connections abruptly closed by sending GOAWAY Hi All, Can you please help us understand this behavior . The following is the sequence of events that is happening for a few streams . Initially when the stream is opened , it processes fine , but beyond some time tomcat does not process the headers, for which the client is sending RST_STREAM (ERROR: NO_ERROR) and then tomcat closes the stream with GOAWAY (An error occurred during processing that was fatal to the connection). Other connection streams during the same time are working fine . Please find the link to the images : img2 has GOAWAY packet https://protect2.fireeye.com/v1/url?k=85ab1ff6-db0bff6f-85ab5f6d-8682aaa22bc0-561cdc1c5bb5b447=1=47308773-70b3-4fac-9ca3-96f35ff17679=https%3A%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F11r6rnbciayHIMNCLXDm4JFUowHJqp8ox Please help us analyze under which cases this happens . Embedded tomcat:9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas mailto:ma...@apache.org>> Sent: Monday, September 7, 2020 10:21 PM To: users@tomcat.apache.org<mailto:users@tomcat.apache.org> Subject: Re: HTTP2: Connections abruptly closed by sending GOAWAY On 07/09/2020 09:29, Arshiya Shariff wrote: > Hi All, > Tomcat is closing connections abruptly by sending GOAWAY with reason > Connection [5309], Stream [57,359], An error occurred during processing that > was fatal to the connection . > > Just trying to understand in what scenarios this happens. Can you please help > us. An unhandled InterruptedIOException during I/O. I/O exception writing an intermediate 100 response for a request with an expectation. I/O exception trying to initiate a push request. I/O exception committing, flushing or closing a stream. So, generally, any I/O exception that indicates the connection between the client and the server is broken. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org> For additional commands, e-mail: users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Any update on 9.0.38 release plan
Hi All, Can we please get a tentative release date for 9.0.38 . Thanks and Regards Arshiya Shariff
RE: Track native memory of a Tomcat application
Hi All, Thank you for the response Christopher . * A single request, or a single *type* of request? A single request (http/https) that is hit once per day * Does it increase as the request is processed, or does the JVM take that 4GiB immediately upon startup? The tomcat process was started 3 months back , but the memory has increased gradually to 4GB though it does not process requests continuously (one request per day). * The first thing to do would be to post the effective command-line options that are being used when launching your JVM. One of the easiest ways to do that is to run "ps | grep catalina" which will give you the full command-line for the process. There is no Xmx set: ogw 30853 0.1 3.1 22048736 2068856 ?Sl Jun09 154:10 /opt/java//jdk1.8.0_201-amd64/bin/java -Djava.util.logging.config.file=/opt/webstart//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=false -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=false -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=false -Dorg.apache.catalina.connector.RECYCLE_FACADES=false -Dignore.endorsed.dirs= -classpath /opt/webstart//bin/bootstrap.jar:/opt/webstart//bin/tomcat-juli.jar -Dcatalina.base=/opt/webstart/ -Dcatalina.home=/opt/webstart/ -Djava.io.tmpdir=/opt/webstart//temp org.apache.catalina.startup.Bootstrap start Thanks and Regards Arshiya Shariff -Original Message- From: Christopher Schultz Sent: Friday, September 11, 2020 10:54 PM To: users@tomcat.apache.org Subject: Re: Track native memory of a Tomcat application -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Arshiya, On 9/11/20 13:06, Arshiya Shariff wrote: > We have a standalone tomcat web application(Version 9.0.22) which runs > on Linux . The application is used to process only a single http > request. A single request, or a single *type* of request? > But the physical memory usage of the application has increased to 4GB > (output from the "top" command of Linux) , of which the heap has only > 16 MB of live data. Does it increase as the request is processed, or does the JVM take that 4GiB immediately upon startup? Remember that the heap is only a part of the JVM's memory usage. Examples of things that don't go into the heap are the native memory used by the JVM itself (thinking of the JVM itself as a usual OS process), stacks for each native thread, compiled Java code produced by the JIT, and various I/O buffers. > Is there a way to track the native memory of a tomcat process ? The native memory of a Java application is very difficult to inspect, etc. unless you are *very* familiar with JVMs, the memory manager being used by your JVM, and the platform and architecture on which the JVM is running. The first thing to do would be to post the effective command-line options that are being used when launching your JVM. One of the easiest ways to do that is to run "ps | grep catalina" which will give you the full command-line for the process. That will show any memory parameters you may have specified on the JVM at launch time, which will definitely affect the amount of native memory your JVM process might end up taking up. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://protect2.fireeye.com/v1/url?k=afccd87f-f17c45e5-afcc98e4-867b36d1634c-0bbd1bba405711b0=1=6321ad9b-27ed-403c-8d47-1699f85d092b=https%3A%2F%2Fwww.enigmail.net%2F iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9bsqoACgkQHPApP6U8 pFhfOA//cb427EP07vqPihuQUNWZtbfRyiDn+9B1AWEXLxKChpomSznUDmZWXs/t w7YE6v6h741M8MiHKCJeUNz537DRxzMMCdoIlFFE48mZ/SFrg6xPgpMr2wQksz5S zyKmInSXRmjKFfv1VUZ1tS6nF8dSxC+iSgC+axyBxU3cywVdJ/hCq8ylgq1mc9F7 JllfyxpQ2cRyioYkJWSR04BVDtXMFlyIhuBat8/OrXAEtsqhXCDXvM3cp8RvcIp6 Sg05MCb3nNM2495xt/m8jc1ffcyho5S4CDD1mfNfTLGG5vie5D81i95ExkPt6Voj 8TTI+Yk9MhauSyeroAX67SVJaTDMPqK3fUMuz+IX31tFo/5YMaQAMIe+XF+LW6Kz EsbSDMRB460j8zAigPIt/9KTSfA8xT8R0t3dMPZytaB01tZZi7atq89QcoR+irrP bFiFmErEcTdtyNZWWFeBQiEy1qZ+0J5GZSr7+6ZUwED20RNeWeZ1czHy/JaAsXMQ kNZqx3iXDWbYq6yTTPdC2tJkl5TST+TW62zdJqpjh7L7hh7P9hJSwjwcDvyOdrXD 0uTaU8j+4b2W7Ex7DPO7XVVaYnhFMRJPc0cgKHXyrlz8/5HOTNDxLafHyPXS7dya ffdyNz0k88OawGKXlWrvDQggi82tVInsEJoJID56qEp7x9tUuyQ= =MuBQ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP2 : PING and GOAWAY sent in the same millisecond
Hi All, The KeepAliveTimeout value is configured to the default value of 20 seconds. So when the connection is idle for 20 seconds, tomcat server(Port:1090) is sending PING followed by GOAWAY within the same millisecond. The client also responds to the PING in the same millisecond, but before that tomcat has closed the connection. Embedded tomcat version-9.0.33 Please find the link to the pcap for reference: https://drive.google.com/file/d/1y9vt9LDw0CeR894TVYSxYSdgCR69wxbQ/view?usp=sharing 1. Is the tomcat behavior correct in sending PING and GOAWAY in the same millisecond ? 2. Should tomcat wait for at least few seconds for PING acknowledgement/response ? 3. Ripple effect will be frequent connection closures as it finally ends up in a memory leak due to connection open and close ( we know this memory leak issue is fixed in 9.0.37) Thanks and Regards Arshiya Shariff
Track native memory of a Tomcat application
Hi All, We have a standalone tomcat web application(Version 9.0.22) which runs on Linux . The application is used to process only a single http request. But the physical memory usage of the application has increased to 4GB (output from the "top" command of Linux) , of which the heap has only 16 MB of live data. Is there a way to track the native memory of a tomcat process ? Thanks and Regards Arshiya Shariff
RE: HTTP2: Connections abruptly closed by sending GOAWAY
Hi All, Can you please help us understand this behavior . The following is the sequence of events that is happening for a few streams . Initially when the stream is opened , it processes fine , but beyond some time tomcat does not process the headers, for which the client is sending RST_STREAM (ERROR: NO_ERROR) and then tomcat closes the stream with GOAWAY (An error occurred during processing that was fatal to the connection). Other connection streams during the same time are working fine . Please find the link to the images : img2 has GOAWAY packet https://drive.google.com/drive/folders/11r6rnbciayHIMNCLXDm4JFUowHJqp8ox Please help us analyze under which cases this happens . Embedded tomcat:9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas mailto:ma...@apache.org>> Sent: Monday, September 7, 2020 10:21 PM To: users@tomcat.apache.org<mailto:users@tomcat.apache.org> Subject: Re: HTTP2: Connections abruptly closed by sending GOAWAY On 07/09/2020 09:29, Arshiya Shariff wrote: > Hi All, > Tomcat is closing connections abruptly by sending GOAWAY with reason > Connection [5309], Stream [57,359], An error occurred during processing that > was fatal to the connection . > > Just trying to understand in what scenarios this happens. Can you please help > us. An unhandled InterruptedIOException during I/O. I/O exception writing an intermediate 100 response for a request with an expectation. I/O exception trying to initiate a push request. I/O exception committing, flushing or closing a stream. So, generally, any I/O exception that indicates the connection between the client and the server is broken. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org> For additional commands, e-mail: users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>
RE: HTTP2: Connections abruptly closed by sending GOAWAY
Thank you for the response Mark . Hi All, Can you please help us understand this behavior . The following is the sequence of events that is happening for a few streams . Initially when the stream is opened , it processes fine , but beyond some time tomcat does not process the headers, for which the client is sending RST_STREAM (ERROR: NO_ERROR) and then tomcat closes the stream with GOAWAY (An error occurred during processing that was fatal to the connection). [cid:image001.jpg@01D686D3.10B6DF90] [cid:image002.jpg@01D686D3.10B6DF90] Please help us analyze under which cases this happens . Embedded tomcat:9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Monday, September 7, 2020 10:21 PM To: users@tomcat.apache.org Subject: Re: HTTP2: Connections abruptly closed by sending GOAWAY On 07/09/2020 09:29, Arshiya Shariff wrote: > Hi All, > Tomcat is closing connections abruptly by sending GOAWAY with reason > Connection [5309], Stream [57,359], An error occurred during processing that > was fatal to the connection . > > Just trying to understand in what scenarios this happens. Can you please help > us. An unhandled InterruptedIOException during I/O. I/O exception writing an intermediate 100 response for a request with an expectation. I/O exception trying to initiate a push request. I/O exception committing, flushing or closing a stream. So, generally, any I/O exception that indicates the connection between the client and the server is broken. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org> For additional commands, e-mail: users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>
HTTP2: Connections abruptly closed by sending GOAWAY
Hi All, Tomcat is closing connections abruptly by sending GOAWAY with reason Connection [5309], Stream [57,359], An error occurred during processing that was fatal to the connection . Just trying to understand in what scenarios this happens. Can you please help us. Embedded tomcat: 9.0.22 Thanks and Regards Arshiya Shariff
RE: Release date of latest Tomcat version - 9.0.38
Hi , Thank you for your response . Is there a way I can track when the version is released for Voting (or) will I get notified in the Users list ? Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, September 2, 2020 12:03 AM To: users@tomcat.apache.org Subject: Re: Release date of latest Tomcat version - 9.0.38 On 01/09/2020 14:01, Christopher Schultz wrote: > Arshiya, > > On 9/1/20 08:13, Arshiya Shariff wrote: >> Hi all, > >> The following reported issue - "HTTP/2 Stream.receivedData method >> throwing continuous NullPointerException in the logs" has been fixed >> in the latest tomcat. >> https://bz.apache.org/bugzilla/show_bug.cgi?id=64671 > >> Can you please share us the release date of tomcat version 9.0.38 . >> We are waiting for the release dates so we can plan accordingly. > There are no promises about release schedule. > > Mark, however, has fairly consistently been rolling releases around > the beginning of each month. If you read the developers list, you'll > see there was a conversation started last week about the next batch of > releases. I'm expecting to tag the next round of releases in a few days. I am currently waiting for the Commons Daemon 1.2.3 release (it was delayed by an issue with the code signing service) and the Tomcat Native 1.2.25 release (vote should hopefully complete shortly). If the Commons Daemon release slips then the Tomcat release will probably slip. > Once a release candidate has been announced (look for [VOTE] threads), > you can help by testing the release: Big +1 > 1. Run the test suite (ant test) > 2. Run the release-candidate on your own application > > Your vote is not binding, but if you find a problem, we will likely > stop the release to fix it. Indeed. We've done that a few times. Much better to test the release during the vote (when we can fix it and re-roll the release) rather than once it is released (when you have to wait for the next release in ~1 month). The number of times people have downloaded a new release and found an issue that they could have caught if they tested during the vote... Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Release date of latest Tomcat version - 9.0.38
Hi all, The following reported issue - "HTTP/2 Stream.receivedData method throwing continuous NullPointerException in the logs" has been fixed in the latest tomcat. https://bz.apache.org/bugzilla/show_bug.cgi?id=64671 Can you please share us the release date of tomcat version 9.0.38 . We are waiting for the release dates so we can plan accordingly. Thanks and Regards Arshiya Shariff
HTTP2:When does tomcat send continuous RST_STREAM with reason REFUSED _STREAM
Hi, What are the cases when tomcat sends RST_STREAM with reason REFUSED_STREAM continuously for every http2 request from the client ? Also , In this case we did not see the max concurrent streams reaching 100. Embedded Tomcat : 9.0.22 Thanks and Regards Arshiya Shariff
HTTP2:When does tomcat send continuous RST_STREAM with reason REFUSED _STREAM
Hi, What are the cases when tomcat sends RST_STREAM with reason REFUSED_STREAM continuously for every http2 request from the client ? Also , In this case we did not see the max concurrent streams reaching 100. Embedded Tomcat : 9.0.22 Thanks and Regards Arshiya Shariff
Is it possible to get a callback notification when a http/http2 connection is opened/closed
Hi All, Can we get a callback notification when a http/http2 connection is opened/closed in Embedded tomcat . Thanks and Regards Arshiya Shariff
HTTP2 keepAliveTimeout changed from Infinite to 20 seconds (default)
Hi Mark, The keepAliveTimeout for HTTP2 , in the later tomcat versions is set to a default value of 20 seconds and can be overridden . Is there a reason that the keepAliveTimeout has been changed from infinite to 20 seconds (default) . Were there any issues around this ? This is just for an information. Thanks and Regards Arshiya Shariff
keepAliveTimeout/connectionTimeout not working with configured values - Embedded Tomcat 9.0.35
Hi Mark , We have upgraded the Embedded tomcat version from 9.0.22 to 9.0.35 and tried setting the keepAliveTimeout/connectionTimeout to 4ms / -1 , via the IntrospectionUtils by extending the Connector class: IntrospectionUtils.setProperty( this, "keepAliveTimeout", String.valueOf( keepAliveTimeout ) ); But we see that the http2 Connection is closing in 20 seconds with a GOAWAY on being idle , irrespective of the values configured . [cid:image001.jpg@01D635A1.DD3149D0] Thanks and Regards Arshiya Shariff
RE: RST on TCP level sent by Tomcat
Hi Mark , 1.Currently we have configured max http2 threads as 40 , but tomcat is allowing more than 300 connections , is there a way to check how many http2 connections tomcat will allow ? 2. Is maxThreads the maxConnections Or is there any other way to set max connections ? We are setting properties by extending the Connector.java class. Embedded Tomcat : 9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, May 20, 2020 3:42 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 20/05/2020 10:07, Arshiya Shariff wrote: > Hi Mark, > Thank you for the response. > > Getting back on Query 3 and 4. > >>>> There are no active streams and still connection is not being closed by >>>> tomcat , and after sometime for new requests tomcat is sending RST. >>>> As it is a production issue, it's hard for us to reproduce this at our end >>>> and retest. > > 1.How long does new connection have to wait when connection limit reached , > when TCP closed it with RST forsuch waiting connections ? That will depend on the client's connection timeout. Tomcat has no control over that. > 2.What is the idle timeout in 9.0.22 for http2 if not provided , will there > be issues if it is infinite also ? Again. You need to upgrade. There issues with HTTP/2 timeouts in that version. Mark > > > Thanks and Regards > Arshiya Shariff > > > -Original Message- > From: Mark Thomas > Sent: Wednesday, May 20, 2020 1:00 PM > To: users@tomcat.apache.org > Subject: Re: RST on TCP level sent by Tomcat > > On 20/05/2020 07:02, Arshiya Shariff wrote: >> Hi Team , >> >> 1.We are facing a problem where tomcat is closing the http2 connections >> silently without sending GOAWAY and FIN. Under what cases does this happen ? > > Tomcat always tries to write the GOAWAY frame. Network issues may prevent the > client receiving it. > >> 2. What happens when maxkeepaliverequests reaches the configured limit, will >> it close connections silently? > > Nothing. The limit does not exist in HTTP/2. > >> 3. What happens when max Connections is reached, will it close older >> connections? > > No. New connections will have to wait until a connection is available. > >> 4. Currently we see keepalive timeout is default 20 seconds, but the >> connection is not closed after that. For requests received after 3 hours >> also we are sending response .Is there any way to close idle-connections ? > > Again, please upgrade and re-test. > > The keep-alive timeout only applies once the entire connection is idle - i.e. > there are no currently processing streams. > > Mark > > >> >> Embedded Tomcat : 9.0.22 >> >> Thanks and Regards >> Arshiya Shariff >> >> >> -Original Message- >> From: Arshiya Shariff >> Sent: Monday, May 18, 2020 4:45 PM >> To: Mark Thomas ; users@tomcat.apache.org >> Cc: M Venkata Pratap M >> Subject: RE: RST on TCP level sent by Tomcat >> >> Hi Mark, >> Thank you for the quick response. >> >> Please provide us a little more clarity on the 3rd query : >> >> 3. We see that RST is sent by tomcat on receiving http2 request, when does >> this happen ? >>>>> When things go wrong. E.g. when the client sends a request to a >>>>> connection that has been closed. >> >> Why does tomcat not send GOAWAY on connection close, upon next request from >> client it sends RST ? >> >> Also, Can you please send us the references to the timeout related fixes in >> 9.0.35 (since 9.0.22). >> >> Thanks and Regards >> Arshiya Shariff >> >> >> >> -Original Message- >> From: Mark Thomas >> Sent: Monday, May 18, 2020 4:17 PM >> To: users@tomcat.apache.org >> Subject: Re: RST on TCP level sent by Tomcat >> >> On 18/05/2020 11:01, Arshiya Shariff wrote: >>> Hi Team, >>> >>> Can you please help us with the below queries : >> >> There have been various timeout related fixes since 9.0.22. Please upgrade >> to 9.0.35 and re-test. >> >>> 1. When does a http2 connection close ? We see that the >>> keepAliveTimeout is >>> 20 seconds by default, but it is not closing the connection on >>> keepAliveTimeout. >> >> Please re-test with 9.0.35. >> >>> 2. How to keep the connections alive / How to enable ping frames to >>> be sent to the other end to keep the connection alive ? >> >> There is
RE: RST on TCP level sent by Tomcat
Hi All, 1. How to set max connections and keepalive timeout in http2 ? The remaining properties are being set through IntrospectionUtils . Embedded Tomcat : 9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, May 20, 2020 3:42 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 20/05/2020 10:07, Arshiya Shariff wrote: > Hi Mark, > Thank you for the response. > > Getting back on Query 3 and 4. > >>>> There are no active streams and still connection is not being closed by >>>> tomcat , and after sometime for new requests tomcat is sending RST. >>>> As it is a production issue, it's hard for us to reproduce this at our end >>>> and retest. > > 1.How long does new connection have to wait when connection limit reached , > when TCP closed it with RST forsuch waiting connections ? That will depend on the client's connection timeout. Tomcat has no control over that. > 2.What is the idle timeout in 9.0.22 for http2 if not provided , will there > be issues if it is infinite also ? Again. You need to upgrade. There issues with HTTP/2 timeouts in that version. Mark > > > Thanks and Regards > Arshiya Shariff > > > -Original Message- > From: Mark Thomas > Sent: Wednesday, May 20, 2020 1:00 PM > To: users@tomcat.apache.org > Subject: Re: RST on TCP level sent by Tomcat > > On 20/05/2020 07:02, Arshiya Shariff wrote: >> Hi Team , >> >> 1.We are facing a problem where tomcat is closing the http2 connections >> silently without sending GOAWAY and FIN. Under what cases does this happen ? > > Tomcat always tries to write the GOAWAY frame. Network issues may prevent the > client receiving it. > >> 2. What happens when maxkeepaliverequests reaches the configured limit, will >> it close connections silently? > > Nothing. The limit does not exist in HTTP/2. > >> 3. What happens when max Connections is reached, will it close older >> connections? > > No. New connections will have to wait until a connection is available. > >> 4. Currently we see keepalive timeout is default 20 seconds, but the >> connection is not closed after that. For requests received after 3 hours >> also we are sending response .Is there any way to close idle-connections ? > > Again, please upgrade and re-test. > > The keep-alive timeout only applies once the entire connection is idle - i.e. > there are no currently processing streams. > > Mark > > >> >> Embedded Tomcat : 9.0.22 >> >> Thanks and Regards >> Arshiya Shariff >> >> >> -Original Message- >> From: Arshiya Shariff >> Sent: Monday, May 18, 2020 4:45 PM >> To: Mark Thomas ; users@tomcat.apache.org >> Cc: M Venkata Pratap M >> Subject: RE: RST on TCP level sent by Tomcat >> >> Hi Mark, >> Thank you for the quick response. >> >> Please provide us a little more clarity on the 3rd query : >> >> 3. We see that RST is sent by tomcat on receiving http2 request, when does >> this happen ? >>>>> When things go wrong. E.g. when the client sends a request to a >>>>> connection that has been closed. >> >> Why does tomcat not send GOAWAY on connection close, upon next request from >> client it sends RST ? >> >> Also, Can you please send us the references to the timeout related fixes in >> 9.0.35 (since 9.0.22). >> >> Thanks and Regards >> Arshiya Shariff >> >> >> >> -Original Message- >> From: Mark Thomas >> Sent: Monday, May 18, 2020 4:17 PM >> To: users@tomcat.apache.org >> Subject: Re: RST on TCP level sent by Tomcat >> >> On 18/05/2020 11:01, Arshiya Shariff wrote: >>> Hi Team, >>> >>> Can you please help us with the below queries : >> >> There have been various timeout related fixes since 9.0.22. Please upgrade >> to 9.0.35 and re-test. >> >>> 1. When does a http2 connection close ? We see that the >>> keepAliveTimeout is >>> 20 seconds by default, but it is not closing the connection on >>> keepAliveTimeout. >> >> Please re-test with 9.0.35. >> >>> 2. How to keep the connections alive / How to enable ping frames to >>> be sent to the other end to keep the connection alive ? >> >> There is no standard API to send an HTTP/2 ping. If you want to keep the >> connections alive for longer, use a longer keep-alive setting. >> >>> 3. We see that RST is sent by tomcat on r
RE: RST on TCP level sent by Tomcat
Hi Mark, Thank you for the response. Getting back on Query 3 and 4. >>> There are no active streams and still connection is not being closed by >>> tomcat , and after sometime for new requests tomcat is sending RST. >>> As it is a production issue, it's hard for us to reproduce this at our end >>> and retest. 1.How long does new connection have to wait when connection limit reached , when TCP closed it with RST forsuch waiting connections ? 2.What is the idle timeout in 9.0.22 for http2 if not provided , will there be issues if it is infinite also ? Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, May 20, 2020 1:00 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 20/05/2020 07:02, Arshiya Shariff wrote: > Hi Team , > > 1.We are facing a problem where tomcat is closing the http2 connections > silently without sending GOAWAY and FIN. Under what cases does this happen ? Tomcat always tries to write the GOAWAY frame. Network issues may prevent the client receiving it. > 2. What happens when maxkeepaliverequests reaches the configured limit, will > it close connections silently? Nothing. The limit does not exist in HTTP/2. > 3. What happens when max Connections is reached, will it close older > connections? No. New connections will have to wait until a connection is available. > 4. Currently we see keepalive timeout is default 20 seconds, but the > connection is not closed after that. For requests received after 3 hours > also we are sending response .Is there any way to close idle-connections ? Again, please upgrade and re-test. The keep-alive timeout only applies once the entire connection is idle - i.e. there are no currently processing streams. Mark > > Embedded Tomcat : 9.0.22 > > Thanks and Regards > Arshiya Shariff > > > -Original Message- > From: Arshiya Shariff > Sent: Monday, May 18, 2020 4:45 PM > To: Mark Thomas ; users@tomcat.apache.org > Cc: M Venkata Pratap M > Subject: RE: RST on TCP level sent by Tomcat > > Hi Mark, > Thank you for the quick response. > > Please provide us a little more clarity on the 3rd query : > > 3. We see that RST is sent by tomcat on receiving http2 request, when does > this happen ? >>>> When things go wrong. E.g. when the client sends a request to a connection >>>> that has been closed. > > Why does tomcat not send GOAWAY on connection close, upon next request from > client it sends RST ? > > Also, Can you please send us the references to the timeout related fixes in > 9.0.35 (since 9.0.22). > > Thanks and Regards > Arshiya Shariff > > > > -Original Message- > From: Mark Thomas > Sent: Monday, May 18, 2020 4:17 PM > To: users@tomcat.apache.org > Subject: Re: RST on TCP level sent by Tomcat > > On 18/05/2020 11:01, Arshiya Shariff wrote: >> Hi Team, >> >> Can you please help us with the below queries : > > There have been various timeout related fixes since 9.0.22. Please upgrade to > 9.0.35 and re-test. > >> 1. When does a http2 connection close ? We see that the >> keepAliveTimeout is >> 20 seconds by default, but it is not closing the connection on >> keepAliveTimeout. > > Please re-test with 9.0.35. > >> 2. How to keep the connections alive / How to enable ping frames to >> be sent to the other end to keep the connection alive ? > > There is no standard API to send an HTTP/2 ping. If you want to keep the > connections alive for longer, use a longer keep-alive setting. > >> 3. We see that RST is sent by tomcat on receiving http2 request, when >> does this happen ? > > When things go wrong. E.g. when the client sends a request to a connection > that has been closed. > >> 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? > > There are no recommended settings. > > Mark > > >> >> >> >> Embedded Tomcat : 9.0.22 >> >> Java Version : 1.8.0.201 >> >> Hardware : Red Hat Enterprise Linux Server release >> 7.4 >> >> >> >> Thanks and Regards >> >> Arshiya Shariff >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: RST on TCP level sent by Tomcat
Hi Mark, Thank you for the response. Getting back on Query 3 and 4. >>> There are no active streams and still connection is not being closed by >>> tomcat , and after sometime for new requests tomcat is sending RST. How long does new connection have to wait when connection limit reached , when TCP closed it with RST forsuch waiting connections ? >>> As it is a production issue, it's hard for us to reproduce this at our end >>> and retest. Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Wednesday, May 20, 2020 1:00 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 20/05/2020 07:02, Arshiya Shariff wrote: > Hi Team , > > 1.We are facing a problem where tomcat is closing the http2 connections > silently without sending GOAWAY and FIN. Under what cases does this happen ? Tomcat always tries to write the GOAWAY frame. Network issues may prevent the client receiving it. > 2. What happens when maxkeepaliverequests reaches the configured limit, will > it close connections silently? Nothing. The limit does not exist in HTTP/2. > 3. What happens when max Connections is reached, will it close older > connections? No. New connections will have to wait until a connection is available. > 4. Currently we see keepalive timeout is default 20 seconds, but the > connection is not closed after that. For requests received after 3 hours > also we are sending response .Is there any way to close idle-connections ? Again, please upgrade and re-test. The keep-alive timeout only applies once the entire connection is idle - i.e. there are no currently processing streams. Mark > > Embedded Tomcat : 9.0.22 > > Thanks and Regards > Arshiya Shariff > > > -Original Message- > From: Arshiya Shariff > Sent: Monday, May 18, 2020 4:45 PM > To: Mark Thomas ; users@tomcat.apache.org > Cc: M Venkata Pratap M > Subject: RE: RST on TCP level sent by Tomcat > > Hi Mark, > Thank you for the quick response. > > Please provide us a little more clarity on the 3rd query : > > 3. We see that RST is sent by tomcat on receiving http2 request, when does > this happen ? >>>> When things go wrong. E.g. when the client sends a request to a connection >>>> that has been closed. > > Why does tomcat not send GOAWAY on connection close, upon next request from > client it sends RST ? > > Also, Can you please send us the references to the timeout related fixes in > 9.0.35 (since 9.0.22). > > Thanks and Regards > Arshiya Shariff > > > > -Original Message- > From: Mark Thomas > Sent: Monday, May 18, 2020 4:17 PM > To: users@tomcat.apache.org > Subject: Re: RST on TCP level sent by Tomcat > > On 18/05/2020 11:01, Arshiya Shariff wrote: >> Hi Team, >> >> Can you please help us with the below queries : > > There have been various timeout related fixes since 9.0.22. Please upgrade to > 9.0.35 and re-test. > >> 1. When does a http2 connection close ? We see that the >> keepAliveTimeout is >> 20 seconds by default, but it is not closing the connection on >> keepAliveTimeout. > > Please re-test with 9.0.35. > >> 2. How to keep the connections alive / How to enable ping frames to >> be sent to the other end to keep the connection alive ? > > There is no standard API to send an HTTP/2 ping. If you want to keep the > connections alive for longer, use a longer keep-alive setting. > >> 3. We see that RST is sent by tomcat on receiving http2 request, when >> does this happen ? > > When things go wrong. E.g. when the client sends a request to a connection > that has been closed. > >> 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? > > There are no recommended settings. > > Mark > > >> >> >> >> Embedded Tomcat : 9.0.22 >> >> Java Version : 1.8.0.201 >> >> Hardware : Red Hat Enterprise Linux Server release >> 7.4 >> >> >> >> Thanks and Regards >> >> Arshiya Shariff >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: RST on TCP level sent by Tomcat
Hi Team , 1.We are facing a problem where tomcat is closing the http2 connections silently without sending GOAWAY and FIN. Under what cases does this happen ? 2. What happens when maxkeepaliverequests reaches the configured limit, will it close connections silently? 3. What happens when max Connections is reached, will it close older connections? 4. Currently we see keepalive timeout is default 20 seconds, but the connection is not closed after that. For requests received after 3 hours also we are sending response .Is there any way to close idle-connections ? Embedded Tomcat : 9.0.22 Thanks and Regards Arshiya Shariff -Original Message- From: Arshiya Shariff Sent: Monday, May 18, 2020 4:45 PM To: Mark Thomas ; users@tomcat.apache.org Cc: M Venkata Pratap M Subject: RE: RST on TCP level sent by Tomcat Hi Mark, Thank you for the quick response. Please provide us a little more clarity on the 3rd query : 3. We see that RST is sent by tomcat on receiving http2 request, when does this happen ? >>> When things go wrong. E.g. when the client sends a request to a connection >>> that has been closed. Why does tomcat not send GOAWAY on connection close, upon next request from client it sends RST ? Also, Can you please send us the references to the timeout related fixes in 9.0.35 (since 9.0.22). Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Monday, May 18, 2020 4:17 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 18/05/2020 11:01, Arshiya Shariff wrote: > Hi Team, > > Can you please help us with the below queries : There have been various timeout related fixes since 9.0.22. Please upgrade to 9.0.35 and re-test. > 1. When does a http2 connection close ? We see that the > keepAliveTimeout is > 20 seconds by default, but it is not closing the connection on > keepAliveTimeout. Please re-test with 9.0.35. > 2. How to keep the connections alive / How to enable ping frames to be > sent to the other end to keep the connection alive ? There is no standard API to send an HTTP/2 ping. If you want to keep the connections alive for longer, use a longer keep-alive setting. > 3. We see that RST is sent by tomcat on receiving http2 request, when > does this happen ? When things go wrong. E.g. when the client sends a request to a connection that has been closed. > 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? There are no recommended settings. Mark > > > > Embedded Tomcat : 9.0.22 > > Java Version : 1.8.0.201 > > Hardware : Red Hat Enterprise Linux Server release > 7.4 > > > > Thanks and Regards > > Arshiya Shariff > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: RST on TCP level sent by Tomcat
Hi Mark, Thank you for the quick response. Please provide us a little more clarity on the 3rd query : 3. We see that RST is sent by tomcat on receiving http2 request, when does this happen ? >>> When things go wrong. E.g. when the client sends a request to a connection >>> that has been closed. Why does tomcat not send GOAWAY on connection close, upon next request from client it sends RST ? Also, Can you please send us the references to the timeout related fixes in 9.0.35 (since 9.0.22). Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Monday, May 18, 2020 4:17 PM To: users@tomcat.apache.org Subject: Re: RST on TCP level sent by Tomcat On 18/05/2020 11:01, Arshiya Shariff wrote: > Hi Team, > > Can you please help us with the below queries : There have been various timeout related fixes since 9.0.22. Please upgrade to 9.0.35 and re-test. > 1. When does a http2 connection close ? We see that the > keepAliveTimeout is > 20 seconds by default, but it is not closing the connection on > keepAliveTimeout. Please re-test with 9.0.35. > 2. How to keep the connections alive / How to enable ping frames to be > sent to the other end to keep the connection alive ? There is no standard API to send an HTTP/2 ping. If you want to keep the connections alive for longer, use a longer keep-alive setting. > 3. We see that RST is sent by tomcat on receiving http2 request, when > does this happen ? When things go wrong. E.g. when the client sends a request to a connection that has been closed. > 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? There are no recommended settings. Mark > > > > Embedded Tomcat : 9.0.22 > > Java Version : 1.8.0.201 > > Hardware : Red Hat Enterprise Linux Server release > 7.4 > > > > Thanks and Regards > > Arshiya Shariff > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RST on TCP level sent by Tomcat
Hi Team, Can you please help us with the below queries : 1. When does a http2 connection close ? We see that the keepAliveTimeout is 20 seconds by default, but it is not closing the connection on keepAliveTimeout. 2. How to keep the connections alive / How to enable ping frames to be sent to the other end to keep the connection alive ? 3. We see that RST is sent by tomcat on receiving http2 request, when does this happen ? 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? Embedded Tomcat : 9.0.22 Java Version : 1.8.0.201 Hardware : Red Hat Enterprise Linux Server release 7.4 Thanks and Regards Arshiya Shariff
RST on TCP level sent by Tomcat
Hi Team, Can you please help us with the below queries : 1. When does a http2 connection close ? We see that the keepAliveTimeout is 20 seconds by default, but it is not closing the connection on keepAliveTimeout. 2. How to keep the connections alive / How to enable ping frames to be sent to the other end to keep the connection alive ? 3. We see that RST is sent by tomcat on receiving http2 request, when does this happen ? 4. What are the recommended ipv4.tcp settings for these kind of scenarios ? Embedded Tomcat : 9.0.22 Java Version : 1.8.0.201 Hardware : Red Hat Enterprise Linux Server release 7.4 Thanks and Regards Arshiya Shariff
