Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Thanks a lot, Mark.

https://ibb.co/LgzFh6t - Memory snapshot after 15minutes of the test.
It's certainly better than the graph with 9.0.36, but I will wait for this
test to run for another few hours. I will update later.

Cheers,
Chirag

On Fri, Jun 26, 2020 at 6:20 PM Mark Thomas  wrote:

> On 26/06/2020 12:48, Mark Thomas wrote:
> > On 26/06/2020 12:45, Chirag Dewan wrote:
> >> Absolutely Mark. Shouldn't take long.
> >
> > Great. I think I have found a potential root cause. If I am right, NIO
> > will show the same issues NIO2 did.
> >
> > I should have a test build for you shortly.
>
> Try this:
> https://people.apache.org/~markt/dev/v9.0.37-dev/
>
> Please note:
>
> This is NOT an official Apache Release.
>
> This build is only intended to be used to test this issue.
>
> Using the test build is it your own risk.
>
> Thanks,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 26/06/2020 12:48, Mark Thomas wrote:
> On 26/06/2020 12:45, Chirag Dewan wrote:
>> Absolutely Mark. Shouldn't take long.
> 
> Great. I think I have found a potential root cause. If I am right, NIO
> will show the same issues NIO2 did.
> 
> I should have a test build for you shortly.

Try this:
https://people.apache.org/~markt/dev/v9.0.37-dev/

Please note:

This is NOT an official Apache Release.

This build is only intended to be used to test this issue.

Using the test build is it your own risk.

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 26/06/2020 12:45, Chirag Dewan wrote:
> Absolutely Mark. Shouldn't take long.

Great. I think I have found a potential root cause. If I am right, NIO
will show the same issues NIO2 did.

I should have a test build for you shortly.

Mark


> 
> On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas,  wrote:
> 
>> Aha!
>>
>> h2c could be the significant factor here. Let me take a look.
>>
>> Are you in a position to test against a dev build if the need arises?
>>
>> Mark
>>
>>
>> On 26/06/2020 11:30, Chirag Dewan wrote:
>>> Hey Mark,
>>>
>>> *Are you using h2c or h2 in your test?*
>>> We are using h2c
>>>
>>>
>>> *Do you see the same issue if you switch the the NIO connector? Note
>>> performance differences between NIO and NIO2 are very small.*
>>>
>>> I havent tried with NIO honestly. Can quickly execute and check. Will
>>> update the results.
>>>
>>> *How long does a single request take to process?*
>>> In normal scenarios, less than 3ms.
>>>
>>> Thanks,
>>> Chirag
>>>
>>> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
>>>
 Hi,

 Thanks for the additional information. The GC roots were particularly
 informative.

 Those RequestInfo objects are associated with HTTP/1.1 requests, not
 HTTP/2 requests.

 Some further questions to try and track down what is going on:

 - Are you using h2c or h2 in your test?

 - Do you see the same issue if you switch the the NIO connector? Note
   performance differences between NIO and NIO2 are very small.

 - How long does a single request take to process?

 Thanks,

 Mark

 On 26/06/2020 09:24, Chirag Dewan wrote:
> Thanks Mark.
>
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate
 as
> many things as possible. Hence a dummy response.
>
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
>> number
> of streams on my client was 500 - which means about 40 connections per
> second.
>
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
>
> I hope I was able to answer everything as expected. Thanks.
>
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
>
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>>>
 On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo
>> objects.

 Thanks. That certainly looks like a memory leak. I'll take a closer
 look. Out of interest, how many threads is the Connector configured
>> to
>> use?

 Mark


>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
 wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
 Hi,

 Update: We found that Tomcat goes OOM when a client closes and
 opens
 new
 connections every second. In the memory dump, we see a lot of
 RequestInfo objects that are causing the memory spike.

 After a while, Tomcat goes OOM and start rejecting request(I
>> get a
 request timed out on my client). This seems like a bug to me.

 For better understanding, let me explain my use case again:

 I have a 

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Absolutely Mark. Shouldn't take long.

On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas,  wrote:

> Aha!
>
> h2c could be the significant factor here. Let me take a look.
>
> Are you in a position to test against a dev build if the need arises?
>
> Mark
>
>
> On 26/06/2020 11:30, Chirag Dewan wrote:
> > Hey Mark,
> >
> > *Are you using h2c or h2 in your test?*
> > We are using h2c
> >
> >
> > *Do you see the same issue if you switch the the NIO connector? Note
> > performance differences between NIO and NIO2 are very small.*
> >
> > I havent tried with NIO honestly. Can quickly execute and check. Will
> > update the results.
> >
> > *How long does a single request take to process?*
> > In normal scenarios, less than 3ms.
> >
> > Thanks,
> > Chirag
> >
> > On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
> >
> >> Hi,
> >>
> >> Thanks for the additional information. The GC roots were particularly
> >> informative.
> >>
> >> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> >> HTTP/2 requests.
> >>
> >> Some further questions to try and track down what is going on:
> >>
> >> - Are you using h2c or h2 in your test?
> >>
> >> - Do you see the same issue if you switch the the NIO connector? Note
> >>   performance differences between NIO and NIO2 are very small.
> >>
> >> - How long does a single request take to process?
> >>
> >> Thanks,
> >>
> >> Mark
> >>
> >> On 26/06/2020 09:24, Chirag Dewan wrote:
> >>> Thanks Mark.
> >>>
> >>> *What is the typical response size for one of these requests? *
> >>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> >>> response in my production use case, but the purpose here was to isolate
> >> as
> >>> many things as possible. Hence a dummy response.
> >>>
> >>> *How long does a typical test take to process? *
> >>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
> number
> >>> of streams on my client was 500 - which means about 40 connections per
> >>> second.
> >>>
> >>> * What are the GC roots for those RequestInfo objects?*
> >>> https://ibb.co/fMRmCXZ
> >>>
> >>> I hope I was able to answer everything as expected. Thanks.
> >>>
> >>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> >>>
>  Thanks.
> 
>  I've looked at the code and I have tried various tests but I am unable
>  to re-create a memory leak.
> 
>  The code used to (before I made a few changes this afternoon) retain a
>  lot more memory per Stream and it is possible that what you are seeing
>  is a system that doesn't have enough memory to achieve steady state.
> 
>  If you are able to build the latest 9.0.x and test that, that could be
>  helpful. Alternatively, I could provide a test build for you to
>  experiment with.
> 
>  Some additional questions that might aid understanding:
> 
>  - What is the typical response size for one of these requests?
>  - How long does a typical test take to process?
>  - What are the GC roots for those RequestInfo objects?
> 
>  Thanks again,
> 
>  Mark
> 
> 
> 
> 
>  On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo
> objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured
> to
>  use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
> >> wrote:
> >>>
>  On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and
> >> opens
> >> new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I
> get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say
> >> 5000)
> >> number of requests/streams and open a new connection that
> >> continues
> 

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

Aha!

h2c could be the significant factor here. Let me take a look.

Are you in a position to test against a dev build if the need arises?

Mark


On 26/06/2020 11:30, Chirag Dewan wrote:
> Hey Mark,
> 
> *Are you using h2c or h2 in your test?*
> We are using h2c
> 
> 
> *Do you see the same issue if you switch the the NIO connector? Note
> performance differences between NIO and NIO2 are very small.*
> 
> I havent tried with NIO honestly. Can quickly execute and check. Will
> update the results.
> 
> *How long does a single request take to process?*
> In normal scenarios, less than 3ms.
> 
> Thanks,
> Chirag
> 
> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:
> 
>> Hi,
>>
>> Thanks for the additional information. The GC roots were particularly
>> informative.
>>
>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>> HTTP/2 requests.
>>
>> Some further questions to try and track down what is going on:
>>
>> - Are you using h2c or h2 in your test?
>>
>> - Do you see the same issue if you switch the the NIO connector? Note
>>   performance differences between NIO and NIO2 are very small.
>>
>> - How long does a single request take to process?
>>
>> Thanks,
>>
>> Mark
>>
>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>> Thanks Mark.
>>>
>>> *What is the typical response size for one of these requests? *
>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>> response in my production use case, but the purpose here was to isolate
>> as
>>> many things as possible. Hence a dummy response.
>>>
>>> *How long does a typical test take to process? *
>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
>>> of streams on my client was 500 - which means about 40 connections per
>>> second.
>>>
>>> * What are the GC roots for those RequestInfo objects?*
>>> https://ibb.co/fMRmCXZ
>>>
>>> I hope I was able to answer everything as expected. Thanks.
>>>
>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
>>>
 Thanks.

 I've looked at the code and I have tried various tests but I am unable
 to re-create a memory leak.

 The code used to (before I made a few changes this afternoon) retain a
 lot more memory per Stream and it is possible that what you are seeing
 is a system that doesn't have enough memory to achieve steady state.

 If you are able to build the latest 9.0.x and test that, that could be
 helpful. Alternatively, I could provide a test build for you to
 experiment with.

 Some additional questions that might aid understanding:

 - What is the typical response size for one of these requests?
 - How long does a typical test take to process?
 - What are the GC roots for those RequestInfo objects?

 Thanks again,

 Mark




 On 25/06/2020 15:10, Chirag Dewan wrote:
> Hi Mark,
>
> Its the default APR connector with 150 Threads.
>
> Chirag
>
> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>
>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>> Thanks for the quick check Mark.
>>>
>>> These are the images I tried referring to:
>>>
>>> https://ibb.co/LzKtRgh
>>>
>>> https://ibb.co/2s7hqRL
>>>
>>> https://ibb.co/KmKj590
>>>
>>>
>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>
>> Thanks. That certainly looks like a memory leak. I'll take a closer
>> look. Out of interest, how many threads is the Connector configured to
 use?
>>
>> Mark
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chirag
>>>
>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
>> wrote:
>>>
 On 24/06/2020 12:17, Mark Thomas wrote:
> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and
>> opens
>> new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me.
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say
>> 5000)
>> number of requests/streams and open a new connection that
>> continues
 to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours
>> my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection
 close
>> exception on my client.
>> On further debugging, I 

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hey Mark,

*Are you using h2c or h2 in your test?*
We are using h2c


*Do you see the same issue if you switch the the NIO connector? Note
performance differences between NIO and NIO2 are very small.*

I havent tried with NIO honestly. Can quickly execute and check. Will
update the results.

*How long does a single request take to process?*
In normal scenarios, less than 3ms.

Thanks,
Chirag

On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas  wrote:

> Hi,
>
> Thanks for the additional information. The GC roots were particularly
> informative.
>
> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> HTTP/2 requests.
>
> Some further questions to try and track down what is going on:
>
> - Are you using h2c or h2 in your test?
>
> - Do you see the same issue if you switch the the NIO connector? Note
>   performance differences between NIO and NIO2 are very small.
>
> - How long does a single request take to process?
>
> Thanks,
>
> Mark
>
> On 26/06/2020 09:24, Chirag Dewan wrote:
> > Thanks Mark.
> >
> > *What is the typical response size for one of these requests? *
> > It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> > response in my production use case, but the purpose here was to isolate
> as
> > many things as possible. Hence a dummy response.
> >
> > *How long does a typical test take to process? *
> > I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> > of streams on my client was 500 - which means about 40 connections per
> > second.
> >
> > * What are the GC roots for those RequestInfo objects?*
> > https://ibb.co/fMRmCXZ
> >
> > I hope I was able to answer everything as expected. Thanks.
> >
> > On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> >
> >> Thanks.
> >>
> >> I've looked at the code and I have tried various tests but I am unable
> >> to re-create a memory leak.
> >>
> >> The code used to (before I made a few changes this afternoon) retain a
> >> lot more memory per Stream and it is possible that what you are seeing
> >> is a system that doesn't have enough memory to achieve steady state.
> >>
> >> If you are able to build the latest 9.0.x and test that, that could be
> >> helpful. Alternatively, I could provide a test build for you to
> >> experiment with.
> >>
> >> Some additional questions that might aid understanding:
> >>
> >> - What is the typical response size for one of these requests?
> >> - How long does a typical test take to process?
> >> - What are the GC roots for those RequestInfo objects?
> >>
> >> Thanks again,
> >>
> >> Mark
> >>
> >>
> >>
> >>
> >> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>> Hi Mark,
> >>>
> >>> Its the default APR connector with 150 Threads.
> >>>
> >>> Chirag
> >>>
> >>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >>>
>  On 25/06/2020 11:00, Chirag Dewan wrote:
> > Thanks for the quick check Mark.
> >
> > These are the images I tried referring to:
> >
> > https://ibb.co/LzKtRgh
> >
> > https://ibb.co/2s7hqRL
> >
> > https://ibb.co/KmKj590
> >
> >
> > The last one is the MAT screenshot showing many RequestInfo objects.
> 
>  Thanks. That certainly looks like a memory leak. I'll take a closer
>  look. Out of interest, how many threads is the Connector configured to
> >> use?
> 
>  Mark
> 
> 
> >
> >
> > Thanks,
> >
> > Chirag
> >
> > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas 
> wrote:
> >
> >> On 24/06/2020 12:17, Mark Thomas wrote:
> >>> On 22/06/2020 11:06, Chirag Dewan wrote:
>  Hi,
> 
>  Update: We found that Tomcat goes OOM when a client closes and
> opens
>  new
>  connections every second. In the memory dump, we see a lot of
>  RequestInfo objects that are causing the memory spike.
> 
>  After a while, Tomcat goes OOM and start rejecting request(I get a
>  request timed out on my client). This seems like a bug to me.
> 
>  For better understanding, let me explain my use case again:
> 
>  I have a jetty client that sends HTTP2 requests to Tomcat. My
>  requirement is to close a connection after a configurable(say
> 5000)
>  number of requests/streams and open a new connection that
> continues
> >> to
>  send requests. I close a connection by sending a GoAway frame.
> 
>  When I execute this use case under load, I see that after ~2hours
> my
>  requests fail and I get a series of errors like request
>  timeouts(5seconds), invalid window update frame, and connection
> >> close
>  exception on my client.
>  On further debugging, I found that it's a Tomcat memory problem
> and
> >> it
>  goes OOM after sometime under heavy load with multiple connections
>  being
>  re-established by the clients.
> 
>  image.png
> 
>  image.png
> 

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

Hi,

Thanks for the additional information. The GC roots were particularly
informative.

Those RequestInfo objects are associated with HTTP/1.1 requests, not
HTTP/2 requests.

Some further questions to try and track down what is going on:

- Are you using h2c or h2 in your test?

- Do you see the same issue if you switch the the NIO connector? Note
  performance differences between NIO and NIO2 are very small.

- How long does a single request take to process?

Thanks,

Mark

On 26/06/2020 09:24, Chirag Dewan wrote:
> Thanks Mark.
> 
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate as
> many things as possible. Hence a dummy response.
> 
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> of streams on my client was 500 - which means about 40 connections per
> second.
> 
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
> 
> I hope I was able to answer everything as expected. Thanks.
> 
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:
> 
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
>>>
 On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo objects.

 Thanks. That certainly looks like a memory leak. I'll take a closer
 look. Out of interest, how many threads is the Connector configured to
>> use?

 Mark


>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
 Hi,

 Update: We found that Tomcat goes OOM when a client closes and opens
 new
 connections every second. In the memory dump, we see a lot of
 RequestInfo objects that are causing the memory spike.

 After a while, Tomcat goes OOM and start rejecting request(I get a
 request timed out on my client). This seems like a bug to me.

 For better understanding, let me explain my use case again:

 I have a jetty client that sends HTTP2 requests to Tomcat. My
 requirement is to close a connection after a configurable(say 5000)
 number of requests/streams and open a new connection that continues
>> to
 send requests. I close a connection by sending a GoAway frame.

 When I execute this use case under load, I see that after ~2hours my
 requests fail and I get a series of errors like request
 timeouts(5seconds), invalid window update frame, and connection
>> close
 exception on my client.
 On further debugging, I found that it's a Tomcat memory problem and
>> it
 goes OOM after sometime under heavy load with multiple connections
 being
 re-established by the clients.

 image.png

 image.png

 Is this a known issue? Or a known behavior with Tomcat?
>>>
>>> Embedded images get dropped by the list software. Post those images
>>> somewhere we can see them.
>>>
 Please let me know if you any experience with such a situation.
>> Thanks
 in advance.
>>>
>>> Nothing comes to mind.
>>>
>>> I'll try some simple tests with HTTP/2.
>>
>> I don't see a memory leak (the memory is reclaimed eventually) but I
>> do
>> see possibilities to release memory associated with request processing
>> sooner.
>>
>> Right now you need to allocate more memory to the Java process to
>> enable
>> Tomcat to handle the HTTP/2 load it is presented with.

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Thanks Mark.

*What is the typical response size for one of these requests? *
It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
response in my production use case, but the purpose here was to isolate as
many things as possible. Hence a dummy response.

*How long does a typical test take to process? *
I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
of streams on my client was 500 - which means about 40 connections per
second.

* What are the GC roots for those RequestInfo objects?*
https://ibb.co/fMRmCXZ

I hope I was able to answer everything as expected. Thanks.

On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas  wrote:

> Thanks.
>
> I've looked at the code and I have tried various tests but I am unable
> to re-create a memory leak.
>
> The code used to (before I made a few changes this afternoon) retain a
> lot more memory per Stream and it is possible that what you are seeing
> is a system that doesn't have enough memory to achieve steady state.
>
> If you are able to build the latest 9.0.x and test that, that could be
> helpful. Alternatively, I could provide a test build for you to
> experiment with.
>
> Some additional questions that might aid understanding:
>
> - What is the typical response size for one of these requests?
> - How long does a typical test take to process?
> - What are the GC roots for those RequestInfo objects?
>
> Thanks again,
>
> Mark
>
>
>
>
> On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured to
> use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
> >>>
>  On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and opens
> >> new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say 5000)
> >> number of requests/streams and open a new connection that continues
> to
> >> send requests. I close a connection by sending a GoAway frame.
> >>
> >> When I execute this use case under load, I see that after ~2hours my
> >> requests fail and I get a series of errors like request
> >> timeouts(5seconds), invalid window update frame, and connection
> close
> >> exception on my client.
> >> On further debugging, I found that it's a Tomcat memory problem and
> it
> >> goes OOM after sometime under heavy load with multiple connections
> >> being
> >> re-established by the clients.
> >>
> >> image.png
> >>
> >> image.png
> >>
> >> Is this a known issue? Or a known behavior with Tomcat?
> >
> > Embedded images get dropped by the list software. Post those images
> > somewhere we can see them.
> >
> >> Please let me know if you any experience with such a situation.
> Thanks
> >> in advance.
> >
> > Nothing comes to mind.
> >
> > I'll try some simple tests with HTTP/2.
> 
>  I don't see a memory leak (the memory is reclaimed eventually) but I
> do
>  see possibilities to release memory associated with request processing
>  sooner.
> 
>  Right now you need to allocate more memory to the Java process to
> enable
>  Tomcat to handle the HTTP/2 load it is presented with.
> 
>  It looks like a reasonable chunk of memory is released when the
>  Connection closes that could be released earlier when the associated
>  Stream closes. I'll take a look at what can be done in that area. In
> the
>  meantime, reducing the number of Streams you allow on a Connection
>  before it is closed should reduce overall memory usage.
> 
>  Mark
> 
>  -
>  To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>  For additional commands, e-mail: 

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

Thanks.

I've looked at the code and I have tried various tests but I am unable
to re-create a memory leak.

The code used to (before I made a few changes this afternoon) retain a
lot more memory per Stream and it is possible that what you are seeing
is a system that doesn't have enough memory to achieve steady state.

If you are able to build the latest 9.0.x and test that, that could be
helpful. Alternatively, I could provide a test build for you to
experiment with.

Some additional questions that might aid understanding:

- What is the typical response size for one of these requests?
- How long does a typical test take to process?
- What are the GC roots for those RequestInfo objects?

Thanks again,

Mark




On 25/06/2020 15:10, Chirag Dewan wrote:
> Hi Mark,
> 
> Its the default APR connector with 150 Threads.
> 
> Chirag
> 
> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:
> 
>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>> Thanks for the quick check Mark.
>>>
>>> These are the images I tried referring to:
>>>
>>> https://ibb.co/LzKtRgh
>>>
>>> https://ibb.co/2s7hqRL
>>>
>>> https://ibb.co/KmKj590
>>>
>>>
>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>
>> Thanks. That certainly looks like a memory leak. I'll take a closer
>> look. Out of interest, how many threads is the Connector configured to use?
>>
>> Mark
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chirag
>>>
>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
>>>
 On 24/06/2020 12:17, Mark Thomas wrote:
> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and opens
>> new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me.
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say 5000)
>> number of requests/streams and open a new connection that continues to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection close
>> exception on my client.
>> On further debugging, I found that it's a Tomcat memory problem and it
>> goes OOM after sometime under heavy load with multiple connections
>> being
>> re-established by the clients.
>>
>> image.png
>>
>> image.png
>>
>> Is this a known issue? Or a known behavior with Tomcat?
>
> Embedded images get dropped by the list software. Post those images
> somewhere we can see them.
>
>> Please let me know if you any experience with such a situation. Thanks
>> in advance.
>
> Nothing comes to mind.
>
> I'll try some simple tests with HTTP/2.

 I don't see a memory leak (the memory is reclaimed eventually) but I do
 see possibilities to release memory associated with request processing
 sooner.

 Right now you need to allocate more memory to the Java process to enable
 Tomcat to handle the HTTP/2 load it is presented with.

 It looks like a reasonable chunk of memory is released when the
 Connection closes that could be released earlier when the associated
 Stream closes. I'll take a look at what can be done in that area. In the
 meantime, reducing the number of Streams you allow on a Connection
 before it is closed should reduce overall memory usage.

 Mark

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org


>>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hi Mark,

Its the default APR connector with 150 Threads.

Chirag

On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas,  wrote:

> On 25/06/2020 11:00, Chirag Dewan wrote:
> > Thanks for the quick check Mark.
> >
> > These are the images I tried referring to:
> >
> > https://ibb.co/LzKtRgh
> >
> > https://ibb.co/2s7hqRL
> >
> > https://ibb.co/KmKj590
> >
> >
> > The last one is the MAT screenshot showing many RequestInfo objects.
>
> Thanks. That certainly looks like a memory leak. I'll take a closer
> look. Out of interest, how many threads is the Connector configured to use?
>
> Mark
>
>
> >
> >
> > Thanks,
> >
> > Chirag
> >
> > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
> >
> >> On 24/06/2020 12:17, Mark Thomas wrote:
> >>> On 22/06/2020 11:06, Chirag Dewan wrote:
>  Hi,
> 
>  Update: We found that Tomcat goes OOM when a client closes and opens
> new
>  connections every second. In the memory dump, we see a lot of
>  RequestInfo objects that are causing the memory spike.
> 
>  After a while, Tomcat goes OOM and start rejecting request(I get a
>  request timed out on my client). This seems like a bug to me.
> 
>  For better understanding, let me explain my use case again:
> 
>  I have a jetty client that sends HTTP2 requests to Tomcat. My
>  requirement is to close a connection after a configurable(say 5000)
>  number of requests/streams and open a new connection that continues to
>  send requests. I close a connection by sending a GoAway frame.
> 
>  When I execute this use case under load, I see that after ~2hours my
>  requests fail and I get a series of errors like request
>  timeouts(5seconds), invalid window update frame, and connection close
>  exception on my client.
>  On further debugging, I found that it's a Tomcat memory problem and it
>  goes OOM after sometime under heavy load with multiple connections
> being
>  re-established by the clients.
> 
>  image.png
> 
>  image.png
> 
>  Is this a known issue? Or a known behavior with Tomcat?
> >>>
> >>> Embedded images get dropped by the list software. Post those images
> >>> somewhere we can see them.
> >>>
>  Please let me know if you any experience with such a situation. Thanks
>  in advance.
> >>>
> >>> Nothing comes to mind.
> >>>
> >>> I'll try some simple tests with HTTP/2.
> >>
> >> I don't see a memory leak (the memory is reclaimed eventually) but I do
> >> see possibilities to release memory associated with request processing
> >> sooner.
> >>
> >> Right now you need to allocate more memory to the Java process to enable
> >> Tomcat to handle the HTTP/2 load it is presented with.
> >>
> >> It looks like a reasonable chunk of memory is released when the
> >> Connection closes that could be released earlier when the associated
> >> Stream closes. I'll take a look at what can be done in that area. In the
> >> meantime, reducing the number of Streams you allow on a Connection
> >> before it is closed should reduce overall memory usage.
> >>
> >> Mark
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
> 
> These are the images I tried referring to:
> 
> https://ibb.co/LzKtRgh
> 
> https://ibb.co/2s7hqRL
> 
> https://ibb.co/KmKj590
> 
> 
> The last one is the MAT screenshot showing many RequestInfo objects.

Thanks. That certainly looks like a memory leak. I'll take a closer
look. Out of interest, how many threads is the Connector configured to use?

Mark


> 
> 
> Thanks,
> 
> Chirag
> 
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:
> 
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
 Hi,

 Update: We found that Tomcat goes OOM when a client closes and opens new
 connections every second. In the memory dump, we see a lot of
 RequestInfo objects that are causing the memory spike.

 After a while, Tomcat goes OOM and start rejecting request(I get a
 request timed out on my client). This seems like a bug to me.

 For better understanding, let me explain my use case again:

 I have a jetty client that sends HTTP2 requests to Tomcat. My
 requirement is to close a connection after a configurable(say 5000)
 number of requests/streams and open a new connection that continues to
 send requests. I close a connection by sending a GoAway frame.

 When I execute this use case under load, I see that after ~2hours my
 requests fail and I get a series of errors like request
 timeouts(5seconds), invalid window update frame, and connection close
 exception on my client.
 On further debugging, I found that it's a Tomcat memory problem and it
 goes OOM after sometime under heavy load with multiple connections being
 re-established by the clients.

 image.png

 image.png

 Is this a known issue? Or a known behavior with Tomcat?
>>>
>>> Embedded images get dropped by the list software. Post those images
>>> somewhere we can see them.
>>>
 Please let me know if you any experience with such a situation. Thanks
 in advance.
>>>
>>> Nothing comes to mind.
>>>
>>> I'll try some simple tests with HTTP/2.
>>
>> I don't see a memory leak (the memory is reclaimed eventually) but I do
>> see possibilities to release memory associated with request processing
>> sooner.
>>
>> Right now you need to allocate more memory to the Java process to enable
>> Tomcat to handle the HTTP/2 load it is presented with.
>>
>> It looks like a reasonable chunk of memory is released when the
>> Connection closes that could be released earlier when the associated
>> Stream closes. I'll take a look at what can be done in that area. In the
>> meantime, reducing the number of Streams you allow on a Connection
>> before it is closed should reduce overall memory usage.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Thanks for the quick check Mark.

These are the images I tried referring to:

https://ibb.co/LzKtRgh

https://ibb.co/2s7hqRL

https://ibb.co/KmKj590


The last one is the MAT screenshot showing many RequestInfo objects.


Thanks,

Chirag

On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas  wrote:

> On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and opens new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say 5000)
> >> number of requests/streams and open a new connection that continues to
> >> send requests. I close a connection by sending a GoAway frame.
> >>
> >> When I execute this use case under load, I see that after ~2hours my
> >> requests fail and I get a series of errors like request
> >> timeouts(5seconds), invalid window update frame, and connection close
> >> exception on my client.
> >> On further debugging, I found that it's a Tomcat memory problem and it
> >> goes OOM after sometime under heavy load with multiple connections being
> >> re-established by the clients.
> >>
> >> image.png
> >>
> >> image.png
> >>
> >> Is this a known issue? Or a known behavior with Tomcat?
> >
> > Embedded images get dropped by the list software. Post those images
> > somewhere we can see them.
> >
> >> Please let me know if you any experience with such a situation. Thanks
> >> in advance.
> >
> > Nothing comes to mind.
> >
> > I'll try some simple tests with HTTP/2.
>
> I don't see a memory leak (the memory is reclaimed eventually) but I do
> see possibilities to release memory associated with request processing
> sooner.
>
> Right now you need to allocate more memory to the Java process to enable
> Tomcat to handle the HTTP/2 load it is presented with.
>
> It looks like a reasonable chunk of memory is released when the
> Connection closes that could be released earlier when the associated
> Stream closes. I'll take a look at what can be done in that area. In the
> meantime, reducing the number of Streams you allow on a Connection
> before it is closed should reduce overall memory usage.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 24/06/2020 12:17, Mark Thomas wrote:
> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and opens new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me. 
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say 5000)
>> number of requests/streams and open a new connection that continues to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection close
>> exception on my client.
>> On further debugging, I found that it's a Tomcat memory problem and it
>> goes OOM after sometime under heavy load with multiple connections being
>> re-established by the clients.
>>
>> image.png
>>
>> image.png
>>
>> Is this a known issue? Or a known behavior with Tomcat?
> 
> Embedded images get dropped by the list software. Post those images
> somewhere we can see them.
> 
>> Please let me know if you any experience with such a situation. Thanks
>> in advance.
> 
> Nothing comes to mind.
> 
> I'll try some simple tests with HTTP/2.

I don't see a memory leak (the memory is reclaimed eventually) but I do
see possibilities to release memory associated with request processing
sooner.

Right now you need to allocate more memory to the Java process to enable
Tomcat to handle the HTTP/2 load it is presented with.

It looks like a reasonable chunk of memory is released when the
Connection closes that could be released earlier when the associated
Stream closes. I'll take a look at what can be done in that area. In the
meantime, reducing the number of Streams you allow on a Connection
before it is closed should reduce overall memory usage.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Mark Thomas]

On 22/06/2020 11:06, Chirag Dewan wrote:
> Hi,
> 
> Update: We found that Tomcat goes OOM when a client closes and opens new
> connections every second. In the memory dump, we see a lot of
> RequestInfo objects that are causing the memory spike.
> 
> After a while, Tomcat goes OOM and start rejecting request(I get a
> request timed out on my client). This seems like a bug to me. 
> 
> For better understanding, let me explain my use case again:
> 
> I have a jetty client that sends HTTP2 requests to Tomcat. My
> requirement is to close a connection after a configurable(say 5000)
> number of requests/streams and open a new connection that continues to
> send requests. I close a connection by sending a GoAway frame.
> 
> When I execute this use case under load, I see that after ~2hours my
> requests fail and I get a series of errors like request
> timeouts(5seconds), invalid window update frame, and connection close
> exception on my client.
> On further debugging, I found that it's a Tomcat memory problem and it
> goes OOM after sometime under heavy load with multiple connections being
> re-established by the clients.
> 
> image.png
> 
> image.png
> 
> Is this a known issue? Or a known behavior with Tomcat?

Embedded images get dropped by the list software. Post those images
somewhere we can see them.

> Please let me know if you any experience with such a situation. Thanks
> in advance.

Nothing comes to mind.

I'll try some simple tests with HTTP/2.

Mark



> 
> On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan  > wrote:
> 
> Hi,
> 
> This is without load balancer actually. I am directly sending to Tomcat.
> 
> Update:
> 
> A part issue I found was to be 9.0.29. I observed that when request
> were timed out on client (2seconds), the client would send a RST
> frame. And the GoAway from Tomcat was perhaps a bug. In 9.0.36, RST
> frame is replied with a RST from Tomcat. 
> 
> Now the next part to troubleshoot is why after about an hour or so,
> requests are timed out at Tomcat.
> 
> Could close to 100 HTTP2 connections per second cause this on Tomcat?
> 
> Thanks
> 
> On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov,  > wrote:
> 
> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> > Hi,
> >
> > We are observing that under high load, my clients start
> receiving a GoAway
> > frame with error:
> >
> > *Connection[{id}], Stream[{id}] an error occurred during
> processing that
> > was fatal to the connection.*
> >
> > Background : We have implemented our clients to close
> connections after
> > every 500-1000 requests (streams). This is a load balancer
> requirement that
> > we are working on and hence such a behavior. So with a
> throughput of around
> > 19k, almost 40 connections are closed and recreated every second.
> >
> > After we receive this frame, my clients start behaving
> erroneously. Before
> > this as well, my clients start sending RST_STREAM with
> canceled for each
> > request. Could this be due to the number of connections we
> open? Is it
> > related to the version of Tomcat? Or maybe my clients are
> misbehaving?
> >
> > Now since I only receive this under heavy load, I can't quite
> picture
> > enough reasons for this to happen.
> >
> > Any possible clues on where I should start looking?
> >
> > My Stack:
> > Server - Tomcat 9.0.29
> > Client - Jetty 9.x
> 
> Does the same happen w/o the load balancer?
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> 
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hi,

Update: We found that Tomcat goes OOM when a client closes and opens new
connections every second. In the memory dump, we see a lot of RequestInfo
objects that are causing the memory spike.

After a while, Tomcat goes OOM and start rejecting request(I get a
request timed out on my client). This seems like a bug to me.

For better understanding, let me explain my use case again:

I have a jetty client that sends HTTP2 requests to Tomcat. My requirement
is to close a connection after a configurable(say 5000) number of
requests/streams and open a new connection that continues to send requests.
I close a connection by sending a GoAway frame.

When I execute this use case under load, I see that after ~2hours my
requests fail and I get a series of errors like request timeouts(5seconds),
invalid window update frame, and connection close exception on my client.
On further debugging, I found that it's a Tomcat memory problem and it goes
OOM after sometime under heavy load with multiple connections being
re-established by the clients.

[image: image.png]

[image: image.png]

Is this a known issue? Or a known behavior with Tomcat?

Please let me know if you any experience with such a situation. Thanks in
advance.

On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan 
wrote:

> Hi,
>
> This is without load balancer actually. I am directly sending to Tomcat.
>
> Update:
>
> A part issue I found was to be 9.0.29. I observed that when request were
> timed out on client (2seconds), the client would send a RST frame. And the
> GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
> a RST from Tomcat.
>
> Now the next part to troubleshoot is why after about an hour or so,
> requests are timed out at Tomcat.
>
> Could close to 100 HTTP2 connections per second cause this on Tomcat?
>
> Thanks
>
> On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, 
> wrote:
>
>> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
>> > Hi,
>> >
>> > We are observing that under high load, my clients start receiving a
>> GoAway
>> > frame with error:
>> >
>> > *Connection[{id}], Stream[{id}] an error occurred during processing that
>> > was fatal to the connection.*
>> >
>> > Background : We have implemented our clients to close connections after
>> > every 500-1000 requests (streams). This is a load balancer requirement
>> that
>> > we are working on and hence such a behavior. So with a throughput of
>> around
>> > 19k, almost 40 connections are closed and recreated every second.
>> >
>> > After we receive this frame, my clients start behaving erroneously.
>> Before
>> > this as well, my clients start sending RST_STREAM with canceled for each
>> > request. Could this be due to the number of connections we open? Is it
>> > related to the version of Tomcat? Or maybe my clients are misbehaving?
>> >
>> > Now since I only receive this under heavy load, I can't quite picture
>> > enough reasons for this to happen.
>> >
>> > Any possible clues on where I should start looking?
>> >
>> > My Stack:
>> > Server - Tomcat 9.0.29
>> > Client - Jetty 9.x
>>
>> Does the same happen w/o the load balancer?
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hi,

This is without load balancer actually. I am directly sending to Tomcat.

Update:

A part issue I found was to be 9.0.29. I observed that when request were
timed out on client (2seconds), the client would send a RST frame. And the
GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
a RST from Tomcat.

Now the next part to troubleshoot is why after about an hour or so,
requests are timed out at Tomcat.

Could close to 100 HTTP2 connections per second cause this on Tomcat?

Thanks

On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov,  wrote:

> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> > Hi,
> >
> > We are observing that under high load, my clients start receiving a
> GoAway
> > frame with error:
> >
> > *Connection[{id}], Stream[{id}] an error occurred during processing that
> > was fatal to the connection.*
> >
> > Background : We have implemented our clients to close connections after
> > every 500-1000 requests (streams). This is a load balancer requirement
> that
> > we are working on and hence such a behavior. So with a throughput of
> around
> > 19k, almost 40 connections are closed and recreated every second.
> >
> > After we receive this frame, my clients start behaving erroneously.
> Before
> > this as well, my clients start sending RST_STREAM with canceled for each
> > request. Could this be due to the number of connections we open? Is it
> > related to the version of Tomcat? Or maybe my clients are misbehaving?
> >
> > Now since I only receive this under heavy load, I can't quite picture
> > enough reasons for this to happen.
> >
> > Any possible clues on where I should start looking?
> >
> > My Stack:
> > Server - Tomcat 9.0.29
> > Client - Jetty 9.x
>
> Does the same happen w/o the load balancer?
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Connection Closure due to Fatal Stream with HTTP2

[Michael Osipov]

Am 2020-06-13 um 08:42 schrieb Chirag Dewan:

Hi,

We are observing that under high load, my clients start receiving a GoAway
frame with error:

*Connection[{id}], Stream[{id}] an error occurred during processing that
was fatal to the connection.*

Background : We have implemented our clients to close connections after
every 500-1000 requests (streams). This is a load balancer requirement that
we are working on and hence such a behavior. So with a throughput of around
19k, almost 40 connections are closed and recreated every second.

After we receive this frame, my clients start behaving erroneously. Before
this as well, my clients start sending RST_STREAM with canceled for each
request. Could this be due to the number of connections we open? Is it
related to the version of Tomcat? Or maybe my clients are misbehaving?

Now since I only receive this under heavy load, I can't quite picture
enough reasons for this to happen.

Any possible clues on where I should start looking?

My Stack:
Server - Tomcat 9.0.29
Client - Jetty 9.x


Does the same happen w/o the load balancer?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Connection Closure due to Fatal Stream with HTTP2

[Chirag Dewan]

Hi,

We are observing that under high load, my clients start receiving a GoAway
frame with error:

*Connection[{id}], Stream[{id}] an error occurred during processing that
was fatal to the connection.*

Background : We have implemented our clients to close connections after
every 500-1000 requests (streams). This is a load balancer requirement that
we are working on and hence such a behavior. So with a throughput of around
19k, almost 40 connections are closed and recreated every second.

After we receive this frame, my clients start behaving erroneously. Before
this as well, my clients start sending RST_STREAM with canceled for each
request. Could this be due to the number of connections we open? Is it
related to the version of Tomcat? Or maybe my clients are misbehaving?

Now since I only receive this under heavy load, I can't quite picture
enough reasons for this to happen.

Any possible clues on where I should start looking?

My Stack:
Server - Tomcat 9.0.29
Client - Jetty 9.x

Thanks in advance