Re: Tomcat 8/NIO performance discrepancies

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 6/12/17 3:43 PM, Piyush Kumar Nayak wrote:
> Sure, I'll check it out and get back. But the last time I tried, 
> JMeter gave me better numbers than ab. Don't want the client to be 
> the bottleneck.

+1

ab evidently doesn't "do" high-concurrency very well. Something about
a single global-lock. Mladen and Jean-Frederic have more information
about it than I do.

- -chris

> -Original Message- From: Mark Thomas
> [mailto:ma...@apache.org] Sent: Tuesday, June 13, 2017 1:05 AM To:
> Tomcat Users List <users@tomcat.apache.org> Subject: Re: Tomcat
> 8/NIO performance discrepancies
> 
> On 12/06/17 20:27, Piyush Kumar Nayak wrote:
>> Thanks, Mark. I have been testing with 100 concurrent users.
>> While the executor had the maxThreads set to 150, the connector
>> without the executor was left to the default value (which should
>> be 200). I did not have the disableKeepAlivePercentage attribute
>> set, which means that it should take to the default value of 75.
>> 
>> So, the number of threads in use before the keepalive is disabled
>> in case of non-executor connector should be 150. At 100 users,
>> the no. of clients is at 50% of the maxThread count. Unless I'm
>> missing something here, I don't see why keep-alive would be
>> disabled. I'd appreciate if you can give me some pointers on how
>> you can tell if requests are processed at the server end with
>> keep-alive.
> 
> I suggest you use ab rather than JMeter. I've seen all sorts of odd
> behaviour with JMeter when processing requests very quickly that I
> haven't seen with ab. Also, ab tells you how many of your requests
> used keep-alive.
> 
> Mark
> 
> 
>> 
>> I'll try setting disableKeepAlivePercentage to 100, and see if
>> that improves the performance.
>> 
>> -----Original Message- From: Mark Thomas
>> [mailto:ma...@apache.org] Sent: Thursday, June 08, 2017 7:13 PM 
>> To: Tomcat Users List <users@tomcat.apache.org> Subject: Re:
>> Tomcat 8/NIO performance discrepancies
>> 
>> On 07/06/17 22:28, Mark Thomas wrote:
>> 
>> 
>> 
>>> I'm still seeing significant differences in BIO with and
>>> without an executor.
>>> 
>>> I'm still looking for the root cause. I have noticed that
>>> without an executor I see far fewer keep-alive requests than
>>> I'd expect. That could explain the difference, or some of it at
>>> least.
>> 
>> Further testing indicates that most, if not all, of the
>> performance difference is down to the fewer number of requests
>> processed using HTTP keep-alive. This is because BIO without an
>> executor is subject to disableKeepAlivePercentage whereas BIO
>> with an executor is not.
>> 
>> If the number of concurrent clients is kept below 75% of
>> maxThreads, performance remains very similar.
>> 
>> The problems you are likely to experience in the real world
>> without disableKeepAlivePercentage (i.e. if you use BIO with an
>> executor) are described in this thread:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomca
t
>>
>> 
.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122
>> aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C6
3
>>
>> 
6325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D&
>> reserved=0
>> 
>> The better tuning options are probably: - increase maxThreads to
>> max expected client concurrency + headroom or if your memory/CPU
>> won't permit that, as high as you can - increase
>> disableKeepAlivePercentage
>> 
>> I'll look at BIO vs NIO next.
>> 
>> Mark
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
-

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Sure, I'll check it out and get back. 
But the last time I tried, JMeter gave me better numbers than ab. Don't want 
the client to be the bottleneck.
 

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Tuesday, June 13, 2017 1:05 AM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies

On 12/06/17 20:27, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> I have been testing with 100 concurrent users. While the executor had the 
> maxThreads set to 150, the connector without the executor was left to the 
> default value (which should be 200). I did not have the 
> disableKeepAlivePercentage attribute set, which means that it should take to 
> the default value of 75.
> 
> So, the number of threads in use before the keepalive is disabled in case of 
> non-executor connector should be 150. At 100 users, the no. of clients is at 
> 50% of the maxThread count. Unless I'm missing something here, I don't see 
> why keep-alive would be disabled.
> I'd appreciate if you can give me some pointers on how you can tell if 
> requests are processed at the server end with keep-alive.

I suggest you use ab rather than JMeter. I've seen all sorts of odd behaviour 
with JMeter when processing requests very quickly that I haven't seen with ab. 
Also, ab tells you how many of your requests used keep-alive.

Mark


> 
> I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
> the performance.
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org]
> Sent: Thursday, June 08, 2017 7:13 PM
> To: Tomcat Users List <users@tomcat.apache.org>
> Subject: Re: Tomcat 8/NIO performance discrepancies
> 
> On 07/06/17 22:28, Mark Thomas wrote:
> 
> 
> 
>> I'm still seeing significant differences in BIO with and without an 
>> executor.
>>
>> I'm still looking for the root cause. I have noticed that without an 
>> executor I see far fewer keep-alive requests than I'd expect. That 
>> could explain the difference, or some of it at least.
> 
> Further testing indicates that most, if not all, of the performance 
> difference is down to the fewer number of requests processed using HTTP 
> keep-alive. This is because BIO without an executor is subject to 
> disableKeepAlivePercentage whereas BIO with an executor is not.
> 
> If the number of concurrent clients is kept below 75% of maxThreads, 
> performance remains very similar.
> 
> The problems you are likely to experience in the real world without 
> disableKeepAlivePercentage (i.e. if you use BIO with an executor) are 
> described in this thread:
> 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat
> .markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122
> aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63
> 6325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D&
> reserved=0
> 
> The better tuning options are probably:
> - increase maxThreads to max expected client concurrency + headroom
>   or if your memory/CPU won't permit that, as high as you can
> - increase disableKeepAlivePercentage
> 
> I'll look at BIO vs NIO next.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Mark,

I apologize for not bringing this up before, but is that BIO vs NIO or BIO with 
executor vs NIO with executor?
Thanks again, for looking into this.

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Friday, June 09, 2017 12:30 AM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies

On 08/06/17 14:42, Mark Thomas wrote:
> I'll look at BIO vs NIO next.

At the sort of request rates and response sizes we are talking about, we are a 
long way from real-world applications and the differences you see between BIO 
and NIO are largely down to the differences between doing a blocking read for 
the next request (BIO) and polling (NIO).

What you will notice is that as the number of concurrent clients rises, BIO 
performance will drop considerably as soon as the disableKeepAlivePercentage is 
exceeded and will continue to fall as client concurrency rises. NIO, on the 
other hand, will retain broadly the same performance even when client 
concurrency >> threads.

I have spent some time looking at this in detail with a profiler but the timing 
differences we are talking about are so small that often the profiler overhead 
ends up dominating.

I think the general lesson here is that benchmarks like this can be interesting 
but the further removed they are from real-world examples, the less useful they 
are likely to be.

As you move towards more real-world examples, what you tend to find is that I/O 
and/or application logic become far more dominant.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 12/06/17 20:27, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> I have been testing with 100 concurrent users. While the executor had the 
> maxThreads set to 150, the connector without the executor was left to the 
> default value (which should be 200). I did not have the 
> disableKeepAlivePercentage attribute set, which means that it should take to 
> the default value of 75.
> 
> So, the number of threads in use before the keepalive is disabled in case of 
> non-executor connector should be 150. At 100 users, the no. of clients is at 
> 50% of the maxThread count. Unless I'm missing something here, I don't see 
> why keep-alive would be disabled.
> I'd appreciate if you can give me some pointers on how you can tell if 
> requests are processed at the server end with keep-alive.

I suggest you use ab rather than JMeter. I've seen all sorts of odd
behaviour with JMeter when processing requests very quickly that I
haven't seen with ab. Also, ab tells you how many of your requests used
keep-alive.

Mark


> 
> I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
> the performance.
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org] 
> Sent: Thursday, June 08, 2017 7:13 PM
> To: Tomcat Users List <users@tomcat.apache.org>
> Subject: Re: Tomcat 8/NIO performance discrepancies
> 
> On 07/06/17 22:28, Mark Thomas wrote:
> 
> 
> 
>> I'm still seeing significant differences in BIO with and without an 
>> executor.
>>
>> I'm still looking for the root cause. I have noticed that without an 
>> executor I see far fewer keep-alive requests than I'd expect. That 
>> could explain the difference, or some of it at least.
> 
> Further testing indicates that most, if not all, of the performance 
> difference is down to the fewer number of requests processed using HTTP 
> keep-alive. This is because BIO without an executor is subject to 
> disableKeepAlivePercentage whereas BIO with an executor is not.
> 
> If the number of concurrent clients is kept below 75% of maxThreads, 
> performance remains very similar.
> 
> The problems you are likely to experience in the real world without 
> disableKeepAlivePercentage (i.e. if you use BIO with an executor) are 
> described in this thread:
> 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D=0
> 
> The better tuning options are probably:
> - increase maxThreads to max expected client concurrency + headroom
>   or if your memory/CPU won't permit that, as high as you can
> - increase disableKeepAlivePercentage
> 
> I'll look at BIO vs NIO next.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Thanks, Mark.
I have been testing with 100 concurrent users. While the executor had the 
maxThreads set to 150, the connector without the executor was left to the 
default value (which should be 200). I did not have the 
disableKeepAlivePercentage attribute set, which means that it should take to 
the default value of 75.

So, the number of threads in use before the keepalive is disabled in case of 
non-executor connector should be 150. At 100 users, the no. of clients is at 
50% of the maxThread count. Unless I'm missing something here, I don't see why 
keep-alive would be disabled.
I'd appreciate if you can give me some pointers on how you can tell if requests 
are processed at the server end with keep-alive.

I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
the performance.

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Thursday, June 08, 2017 7:13 PM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies

On 07/06/17 22:28, Mark Thomas wrote:



> I'm still seeing significant differences in BIO with and without an 
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an 
> executor I see far fewer keep-alive requests than I'd expect. That 
> could explain the difference, or some of it at least.

Further testing indicates that most, if not all, of the performance difference 
is down to the fewer number of requests processed using HTTP keep-alive. This 
is because BIO without an executor is subject to disableKeepAlivePercentage 
whereas BIO with an executor is not.

If the number of concurrent clients is kept below 75% of maxThreads, 
performance remains very similar.

The problems you are likely to experience in the real world without 
disableKeepAlivePercentage (i.e. if you use BIO with an executor) are described 
in this thread:

https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D=0

The better tuning options are probably:
- increase maxThreads to max expected client concurrency + headroom
  or if your memory/CPU won't permit that, as high as you can
- increase disableKeepAlivePercentage

I'll look at BIO vs NIO next.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[kmaxwilliams43]

Envoyé de mon smartphone BlackBerry 10.
  Message d'origine  
De: kmaxwilliam...@gmail.com
Envoyé: vendredi 9 juin 2017 00:12
À: Astrat Goe; Tomcat Users List
Objet: Re: Tomcat 8/NIO performance discrepancies

Gvhuf

Envoyé de mon smartphone BlackBerry 10.
  Message d'origine  
De: Astrat Goe
Envoyé: mercredi 7 juin 2017 21:29
À: Tomcat Users List
Répondre à: Tomcat Users List
Objet: Re: Tomcat 8/NIO performance discrepancies
Ggfggthgd
Vid
‎
> On Jun 8, 2017, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote:
> 
> On 07/06/17 20:25, Piyush Kumar Nayak wrote:
>> Mark,
>> 
>> It may not be important, but just to be clear, I'm not testing on localhost. 
>> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
>> between them.
> 
> I'm aware you were using two machines but that doesn't appear to be a
> significant factor in the observed differences.
> 
>> Can you also please provide some pointers on the following:
>> - How did you disable session on your JSP?
> 
> Look in the index.jsp for the ROOT web application that ships with
> Tomcat for an example. Full details are in the JSP specification.
> 
>> - Did you observe anything that suggests that the client is not closing the 
>> connections?
> 
> Yes. This looks like a JMeter issue. It isn't the first time JMeter has
> given strange results under load. I've switched to ab which I have found
> to be generally much more reliable for these sorts of tests.
> 
> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.
> 
> Mark
> 
> 
>> I did not specify the implementation for the HTTP sampler that I configured 
>> with JMeter, and it is not set in my jmeter.properties file. So, going by 
>> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
>> use.
>> 
>> Thanks
>> Piyush.
>> 
>> -----Original Message-
>> From: Mark Thomas [mailto:ma...@apache.org] 
>> Sent: Wednesday, June 07, 2017 8:27 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Tomcat 8/NIO performance discrepancies
>> 
>> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>>> Thanks, Mark.
>>> 
>>> Here are some additional details.
>> 
>> Thanks. These help a lot to make sure we are testing (at least roughly) the 
>> same thing.
>> 
>>> I am using Apache JMeter to inject load. I am using a simple hello-world 
>>> JSP.
>>> 
>>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
>> 
>> I've disabled session in the JSP for my test and used a simpler JSP.
>> That should amplify any difference in connector performance.
>> 
>> I'm also testing on localhost.
>> 
>>> I have disabled access log in tomcat. All the other server.xml settings are 
>>> the default.
>>> 
>>> 
>>> The BIO with Executor configuration we are using is:
>>> >> maxThreads="150" minSpareThreads="4"/>
>>> 
>>> >> connectionTimeout="2"
>>> redirectPort="8443" />
>> 
>> FYI for anyone following along. That is a non-executor config. The connector 
>> needs to specify the executor it wants to used.
>> 
>> I do see odd behaviour with BIO + executor. It locks up very easily. It 
>> appears that the client isn't closing the connections. I don't see this 
>> problem with BIO but that may be related to how BIO without an executor 
>> handles large numbers of connections.
>> 
>> If I lower the client thread count, the odd behaviour stops and I do see 
>> notably higher throughput with BIO + executor. That is unexpected.
>> 
>> My initial thoughts are maybe a contention issue related to the thread pool. 
>> I'm continuing to investigate. I plan to look at performance first and then 
>> the locking up.
>> 
>> Mark
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[kmaxwilliams43]

Gvhuf

Envoyé de mon smartphone BlackBerry 10.
  Message d'origine  
De: Astrat Goe
Envoyé: mercredi 7 juin 2017 21:29
À: Tomcat Users List
Répondre à: Tomcat Users List
Objet: Re: Tomcat 8/NIO performance discrepancies


> On Jun 8, 2017, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote:
> 
> On 07/06/17 20:25, Piyush Kumar Nayak wrote:
>> Mark,
>> 
>> It may not be important, but just to be clear, I'm not testing on localhost. 
>> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
>> between them.
> 
> I'm aware you were using two machines but that doesn't appear to be a
> significant factor in the observed differences.
> 
>> Can you also please provide some pointers on the following:
>> - How did you disable session on your JSP?
> 
> Look in the index.jsp for the ROOT web application that ships with
> Tomcat for an example. Full details are in the JSP specification.
> 
>> - Did you observe anything that suggests that the client is not closing the 
>> connections?
> 
> Yes. This looks like a JMeter issue. It isn't the first time JMeter has
> given strange results under load. I've switched to ab which I have found
> to be generally much more reliable for these sorts of tests.
> 
> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.
> 
> Mark
> 
> 
>> I did not specify the implementation for the HTTP sampler that I configured 
>> with JMeter, and it is not set in my jmeter.properties file. So, going by 
>> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
>> use.
>> 
>> Thanks
>> Piyush.
>> 
>> -----Original Message-
>> From: Mark Thomas [mailto:ma...@apache.org] 
>> Sent: Wednesday, June 07, 2017 8:27 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Tomcat 8/NIO performance discrepancies
>> 
>> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>>> Thanks, Mark.
>>> 
>>> Here are some additional details.
>> 
>> Thanks. These help a lot to make sure we are testing (at least roughly) the 
>> same thing.
>> 
>>> I am using Apache JMeter to inject load. I am using a simple hello-world 
>>> JSP.
>>> 
>>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
>> 
>> I've disabled session in the JSP for my test and used a simpler JSP.
>> That should amplify any difference in connector performance.
>> 
>> I'm also testing on localhost.
>> 
>>> I have disabled access log in tomcat. All the other server.xml settings are 
>>> the default.
>>> 
>>> 
>>> The BIO with Executor configuration we are using is:
>>> >> maxThreads="150" minSpareThreads="4"/>
>>> 
>>> >> connectionTimeout="2"
>>> redirectPort="8443" />
>> 
>> FYI for anyone following along. That is a non-executor config. The connector 
>> needs to specify the executor it wants to used.
>> 
>> I do see odd behaviour with BIO + executor. It locks up very easily. It 
>> appears that the client isn't closing the connections. I don't see this 
>> problem with BIO but that may be related to how BIO without an executor 
>> handles large numbers of connections.
>> 
>> If I lower the client thread count, the odd behaviour stops and I do see 
>> notably higher throughput with BIO + executor. That is unexpected.
>> 
>> My initial thoughts are maybe a contention issue related to the thread pool. 
>> I'm continuing to investigate. I plan to look at performance first and then 
>> the locking up.
>> 
>> Mark
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 08/06/17 14:42, Mark Thomas wrote:
> I'll look at BIO vs NIO next.

At the sort of request rates and response sizes we are talking about, we
are a long way from real-world applications and the differences you see
between BIO and NIO are largely down to the differences between doing a
blocking read for the next request (BIO) and polling (NIO).

What you will notice is that as the number of concurrent clients rises,
BIO performance will drop considerably as soon as the
disableKeepAlivePercentage is exceeded and will continue to fall as
client concurrency rises. NIO, on the other hand, will retain broadly
the same performance even when client concurrency >> threads.

I have spent some time looking at this in detail with a profiler but the
timing differences we are talking about are so small that often the
profiler overhead ends up dominating.

I think the general lesson here is that benchmarks like this can be
interesting but the further removed they are from real-world examples,
the less useful they are likely to be.

As you move towards more real-world examples, what you tend to find is
that I/O and/or application logic become far more dominant.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 07/06/17 22:28, Mark Thomas wrote:



> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.

Further testing indicates that most, if not all, of the performance
difference is down to the fewer number of requests processed using HTTP
keep-alive. This is because BIO without an executor is subject to
disableKeepAlivePercentage whereas BIO with an executor is not.

If the number of concurrent clients is kept below 75% of maxThreads,
performance remains very similar.

The problems you are likely to experience in the real world without
disableKeepAlivePercentage (i.e. if you use BIO with an executor) are
described in this thread:

http://tomcat.markmail.org/thread/mcfdk2273v2fsh5k

The better tuning options are probably:
- increase maxThreads to max expected client concurrency + headroom
  or if your memory/CPU won't permit that, as high as you can
- increase disableKeepAlivePercentage

I'll look at BIO vs NIO next.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Astrat Goe]

> On Jun 8, 2017, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote:
> 
> On 07/06/17 20:25, Piyush Kumar Nayak wrote:
>> Mark,
>> 
>> It may not be important, but just to be clear, I'm not testing on localhost. 
>> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
>> between them.
> 
> I'm aware you were using two machines but that doesn't appear to be a
> significant factor in the observed differences.
> 
>> Can you also please provide some pointers on the following:
>> - How did you disable session on your JSP?
> 
> Look in the index.jsp for the ROOT web application that ships with
> Tomcat for an example. Full details are in the JSP specification.
> 
>> - Did you observe anything that suggests that the client is not closing the 
>> connections?
> 
> Yes. This looks like a JMeter issue. It isn't the first time JMeter has
> given strange results under load. I've switched to ab which I have found
> to be generally much more reliable for these sorts of tests.
> 
> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.
> 
> Mark
> 
> 
>> I did not specify the implementation for the HTTP sampler that I configured 
>> with JMeter, and it is not set in my jmeter.properties file. So, going by 
>> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
>> use.
>> 
>> Thanks
>> Piyush.
>> 
>> -----Original Message-
>> From: Mark Thomas [mailto:ma...@apache.org] 
>> Sent: Wednesday, June 07, 2017 8:27 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Tomcat 8/NIO performance discrepancies
>> 
>> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>>> Thanks, Mark.
>>> 
>>> Here are some additional details.
>> 
>> Thanks. These help a lot to make sure we are testing (at least roughly) the 
>> same thing.
>> 
>>> I am using Apache JMeter to inject load. I am using a simple hello-world 
>>> JSP.
>>> 
>>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
>> 
>> I've disabled session in the JSP for my test and used a simpler JSP.
>> That should amplify any difference in connector performance.
>> 
>> I'm also testing on localhost.
>> 
>>> I have disabled access log in tomcat. All the other server.xml settings are 
>>> the default.
>>> 
>>> 
>>> The BIO with Executor configuration we are using is:
>>> >>maxThreads="150" minSpareThreads="4"/>
>>> 
>>> >>   connectionTimeout="2"
>>>   redirectPort="8443" />
>> 
>> FYI for anyone following along. That is a non-executor config. The connector 
>> needs to specify the executor it wants to used.
>> 
>> I do see odd behaviour with BIO + executor. It locks up very easily. It 
>> appears that the client isn't closing the connections. I don't see this 
>> problem with BIO but that may be related to how BIO without an executor 
>> handles large numbers of connections.
>> 
>> If I lower the client thread count, the odd behaviour stops and I do see 
>> notably higher throughput with BIO + executor. That is unexpected.
>> 
>> My initial thoughts are maybe a contention issue related to the thread pool. 
>> I'm continuing to investigate. I plan to look at performance first and then 
>> the locking up.
>> 
>> Mark
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Astrat Goe]

> On Jun 8, 2017, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote:
> 
> On 07/06/17 20:25, Piyush Kumar Nayak wrote:
>> Mark,
>> 
>> It may not be important, but just to be clear, I'm not testing on localhost. 
>> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
>> between them.
> 
> I'm aware you were using two machines but that doesn't appear to be a
> significant factor in the observed differences.
> 
>> Can you also please provide some pointers on the following:
>> - How did you disable session on your JSP?
> 
> Look in the index.jsp for the ROOT web application that ships with
> Tomcat for an example. Full details are in the JSP specification.
> 
>> - Did you observe anything that suggests that the client is not closing the 
>> connections?
> 
> Yes. This looks like a JMeter issue. It isn't the first time JMeter has
> given strange results under load. I've switched to ab which I have found
> to be generally much more reliable for these sorts of tests.
> 
> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.
> 
> Mark
> 
> 
>> I did not specify the implementation for the HTTP sampler that I configured 
>> with JMeter, and it is not set in my jmeter.properties file. So, going by 
>> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
>> use.
>> 
>> Thanks
>> Piyush.
>> 
>> -----Original Message-
>> From: Mark Thomas [mailto:ma...@apache.org] 
>> Sent: Wednesday, June 07, 2017 8:27 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Tomcat 8/NIO performance discrepancies
>> 
>> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>>> Thanks, Mark.
>>> 
>>> Here are some additional details.
>> 
>> Thanks. These help a lot to make sure we are testing (at least roughly) the 
>> same thing.
>> 
>>> I am using Apache JMeter to inject load. I am using a simple hello-world 
>>> JSP.
>>> 
>>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
>> 
>> I've disabled session in the JSP for my test and used a simpler JSP.
>> That should amplify any difference in connector performance.
>> 
>> I'm also testing on localhost.
>> 
>>> I have disabled access log in tomcat. All the other server.xml settings are 
>>> the default.
>>> 
>>> 
>>> The BIO with Executor configuration we are using is:
>>> >>maxThreads="150" minSpareThreads="4"/>
>>> 
>>> >>   connectionTimeout="2"
>>>   redirectPort="8443" />
>> 
>> FYI for anyone following along. That is a non-executor config. The connector 
>> needs to specify the executor it wants to used.
>> 
>> I do see odd behaviour with BIO + executor. It locks up very easily. It 
>> appears that the client isn't closing the connections. I don't see this 
>> problem with BIO but that may be related to how BIO without an executor 
>> handles large numbers of connections.
>> 
>> If I lower the client thread count, the odd behaviour stops and I do see 
>> notably higher throughput with BIO + executor. That is unexpected.
>> 
>> My initial thoughts are maybe a contention issue related to the thread pool. 
>> I'm continuing to investigate. I plan to look at performance first and then 
>> the locking up.
>> 
>> Mark
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Astrat Goe]

> On Jun 8, 2017, at 4:28 AM, Mark Thomas <ma...@apache.org> wrote:
> 
> On 07/06/17 20:25, Piyush Kumar Nayak wrote:
>> Mark,
>> 
>> It may not be important, but just to be clear, I'm not testing on localhost. 
>> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
>> between them.
> 
> I'm aware you were using two machines but that doesn't appear to be a
> significant factor in the observed differences.
> 
>> Can you also please provide some pointers on the following:
>> - How did you disable session on your JSP?
> 
> Look in the index.jsp for the ROOT web application that ships with
> Tomcat for an example. Full details are in the JSP specification.
> 
>> - Did you observe anything that suggests that the client is not closing the 
>> connections?
> 
> Yes. This looks like a JMeter issue. It isn't the first time JMeter has
> given strange results under load. I've switched to ab which I have found
> to be generally much more reliable for these sorts of tests.
> 
> I'm still seeing significant differences in BIO with and without an
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an
> executor I see far fewer keep-alive requests than I'd expect. That could
> explain the difference, or some of it at least.
> 
> Mark
> 
> 
>> I did not specify the implementation for the HTTP sampler that I configured 
>> with JMeter, and it is not set in my jmeter.properties file. So, going by 
>> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
>> use.
>> 
>> Thanks
>> Piyush.
>> 
>> -----Original Message-
>> From: Mark Thomas [mailto:ma...@apache.org] 
>> Sent: Wednesday, June 07, 2017 8:27 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Tomcat 8/NIO performance discrepancies
>> 
>> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>>> Thanks, Mark.
>>> 
>>> Here are some additional details.
>> 
>> Thanks. These help a lot to make sure we are testing (at least roughly) the 
>> same thing.
>> 
>>> I am using Apache JMeter to inject load. I am using a simple hello-world 
>>> JSP.
>>> 
>>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
>> 
>> I've disabled session in the JSP for my test and used a simpler JSP.
>> That should amplify any difference in connector performance.
>> 
>> I'm also testing on localhost.
>> 
>>> I have disabled access log in tomcat. All the other server.xml settings are 
>>> the default.
>>> 
>>> 
>>> The BIO with Executor configuration we are using is:
>>> >>maxThreads="150" minSpareThreads="4"/>
>>> 
>>> >>   connectionTimeout="2"
>>>   redirectPort="8443" />
>> 
>> FYI for anyone following along. That is a non-executor config. The connector 
>> needs to specify the executor it wants to used.
>> 
>> I do see odd behaviour with BIO + executor. It locks up very easily. It 
>> appears that the client isn't closing the connections. I don't see this 
>> problem with BIO but that may be related to how BIO without an executor 
>> handles large numbers of connections.
>> 
>> If I lower the client thread count, the odd behaviour stops and I do see 
>> notably higher throughput with BIO + executor. That is unexpected.
>> 
>> My initial thoughts are maybe a contention issue related to the thread pool. 
>> I'm continuing to investigate. I plan to look at performance first and then 
>> the locking up.
>> 
>> Mark
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 07/06/17 20:25, Piyush Kumar Nayak wrote:
> Mark,
> 
> It may not be important, but just to be clear, I'm not testing on localhost. 
> My JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces 
> between them.

I'm aware you were using two machines but that doesn't appear to be a
significant factor in the observed differences.

> Can you also please provide some pointers on the following:
> - How did you disable session on your JSP?

Look in the index.jsp for the ROOT web application that ships with
Tomcat for an example. Full details are in the JSP specification.

> - Did you observe anything that suggests that the client is not closing the 
> connections?

Yes. This looks like a JMeter issue. It isn't the first time JMeter has
given strange results under load. I've switched to ab which I have found
to be generally much more reliable for these sorts of tests.

I'm still seeing significant differences in BIO with and without an
executor.

I'm still looking for the root cause. I have noticed that without an
executor I see far fewer keep-alive requests than I'd expect. That could
explain the difference, or some of it at least.

Mark


> I did not specify the implementation for the HTTP sampler that I configured 
> with JMeter, and it is not set in my jmeter.properties file. So, going by 
> JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in 
> use.
> 
> Thanks
> Piyush.
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org] 
> Sent: Wednesday, June 07, 2017 8:27 PM
> To: Tomcat Users List <users@tomcat.apache.org>
> Subject: Re: Tomcat 8/NIO performance discrepancies
> 
> On 06/06/17 09:10, Piyush Kumar Nayak wrote:
>> Thanks, Mark.
>>
>> Here are some additional details.
> 
> Thanks. These help a lot to make sure we are testing (at least roughly) the 
> same thing.
> 
>> I am using Apache JMeter to inject load. I am using a simple hello-world JSP.
>>
>> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
>> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.
> 
> I've disabled session in the JSP for my test and used a simpler JSP.
> That should amplify any difference in connector performance.
> 
> I'm also testing on localhost.
> 
>> I have disabled access log in tomcat. All the other server.xml settings are 
>> the default.
>>
>>
>> The BIO with Executor configuration we are using is:
>> > maxThreads="150" minSpareThreads="4"/>
>>
>> >connectionTimeout="2"
>>redirectPort="8443" />
> 
> FYI for anyone following along. That is a non-executor config. The connector 
> needs to specify the executor it wants to used.
> 
> I do see odd behaviour with BIO + executor. It locks up very easily. It 
> appears that the client isn't closing the connections. I don't see this 
> problem with BIO but that may be related to how BIO without an executor 
> handles large numbers of connections.
> 
> If I lower the client thread count, the odd behaviour stops and I do see 
> notably higher throughput with BIO + executor. That is unexpected.
> 
> My initial thoughts are maybe a contention issue related to the thread pool. 
> I'm continuing to investigate. I plan to look at performance first and then 
> the locking up.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Mark,

It may not be important, but just to be clear, I'm not testing on localhost. My 
JMeter and Tomcat are on different machines, with 1 Gbps NIC interfaces between 
them.

Can you also please provide some pointers on the following:
- How did you disable session on your JSP?
- Did you observe anything that suggests that the client is not closing the 
connections?
I did not specify the implementation for the HTTP sampler that I configured 
with JMeter, and it is not set in my jmeter.properties file. So, going by 
JMeter's documentation, Apache HttpComponents HttpClient 4.x. should be in use.

Thanks
Piyush.

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Wednesday, June 07, 2017 8:27 PM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies

On 06/06/17 09:10, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> 
> Here are some additional details.

Thanks. These help a lot to make sure we are testing (at least roughly) the 
same thing.

> I am using Apache JMeter to inject load. I am using a simple hello-world JSP.
> 
> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.

I've disabled session in the JSP for my test and used a simpler JSP.
That should amplify any difference in connector performance.

I'm also testing on localhost.

> I have disabled access log in tomcat. All the other server.xml settings are 
> the default.
> 
> 
> The BIO with Executor configuration we are using is:
>  maxThreads="150" minSpareThreads="4"/>
> 
> connectionTimeout="2"
>redirectPort="8443" />

FYI for anyone following along. That is a non-executor config. The connector 
needs to specify the executor it wants to used.

I do see odd behaviour with BIO + executor. It locks up very easily. It appears 
that the client isn't closing the connections. I don't see this problem with 
BIO but that may be related to how BIO without an executor handles large 
numbers of connections.

If I lower the client thread count, the odd behaviour stops and I do see 
notably higher throughput with BIO + executor. That is unexpected.

My initial thoughts are maybe a contention issue related to the thread pool. 
I'm continuing to investigate. I plan to look at performance first and then the 
locking up.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 7 June 2017 17:33:02 BST, Christopher Schultz  
wrote:
>Mark,
>
>On 6/7/17 10:57 AM, Mark Thomas wrote:
>> I do see odd behaviour with BIO + executor. It locks up very
>> easily. It appears that the client isn't closing the connections. I
>> don't see this problem with BIO but that may be related to how BIO
>> without an executor handles large numbers of connections.
>
>Was one of those BIOs supposed to be an "NIO"?

No. The second BIO should have been BIO without an executor.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 6/7/17 10:57 AM, Mark Thomas wrote:
> I do see odd behaviour with BIO + executor. It locks up very
> easily. It appears that the client isn't closing the connections. I
> don't see this problem with BIO but that may be related to how BIO
> without an executor handles large numbers of connections.

Was one of those BIOs supposed to be an "NIO"?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJZOCq+AAoJEBzwKT+lPKRYguwQAIgksfcD9loUKPQoR5LZsjgg
0VNrWVSmn5o4enqjIA5RLOt55XzflK7H4drBN2IB/QfJ/5Z40d9HYAHh9x5xdFst
kyY+3kFveS0GG8lxcutsl2sr2zetV9D7/vYsjfBcvgTB+5SPcwlL9oQjk0SKSO1I
lA6pQCTdEvNfXl+gYuY9SJnDQz6SXxsLXGjUNu7CN5xesjoTGyDdX43DwhGRmrRC
rn00X2sYOlELr2jQ6K7jEPKNM0O2gkIBqlbrYOMY4hbas8K9UI+m4apsk+jPoFeG
iSGUIl7bgLIymawFyWcjhAjkjm6qrzEorBkKbO+HqSr85SsvtyfCax39RmtzqTR/
VQZka/0NFSO57pScAeWz1hZxC92Iyxj6o3LLtWtm2vWFzyw8t9QhSMvtHxok5Moj
hENhDaz+bOv5s+yWjrvWJB7zHOD8mmjrZka8tmvTdcqoESFggXG1yYPwb3NrPHoc
p7/Sy97g21t8lCbPV2LUfa8mpEf0L6I98SpwxJ3dGuXS3r9FqdsTPPEIOXjD/fsv
/0GkupemYMk9AycIbCapK1xklj3qUDKuCsePtl3A/InZlPxiKMhQ357xz/sH1AT7
ZPCoItvp+gsEGb01moIX1TwaxQ8ubiXtHYgXAAAHhVeMUIrEmuW8b7nC4DXyMJUe
cmSZQy23IkBUaqVDcY/d
=gPB6
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 06/06/17 09:10, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> 
> Here are some additional details.

Thanks. These help a lot to make sure we are testing (at least roughly)
the same thing.

> I am using Apache JMeter to inject load. I am using a simple hello-world JSP.
> 
> The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs 
> and uses an HTTP Cookie Manager, that reuses cookies for each user thread.

I've disabled session in the JSP for my test and used a simpler JSP.
That should amplify any difference in connector performance.

I'm also testing on localhost.

> I have disabled access log in tomcat. All the other server.xml settings are 
> the default.
> 
> 
> The BIO with Executor configuration we are using is:
>  maxThreads="150" minSpareThreads="4"/>
> 
> connectionTimeout="2"
>redirectPort="8443" />

FYI for anyone following along. That is a non-executor config. The
connector needs to specify the executor it wants to used.

I do see odd behaviour with BIO + executor. It locks up very easily. It
appears that the client isn't closing the connections. I don't see this
problem with BIO but that may be related to how BIO without an executor
handles large numbers of connections.

If I lower the client thread count, the odd behaviour stops and I do see
notably higher throughput with BIO + executor. That is unexpected.

My initial thoughts are maybe a contention issue related to the thread
pool. I'm continuing to investigate. I plan to look at performance first
and then the locking up.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Just realized that the comparison matrix formatting was lost earlier, so 
sharing it again.

-
helloWorld.JSP  BIO NIO BIO w/ Exec 
NIO w/ Exec 
Throughput ART  Throughput  ART Throughput ART  Throughput ART
-
   
Tomcat 7.07738600   2   42664   257104  
   143660   2
Tomcat 8.0.44   38585   2   44752   263000  
   144341   2
Tomcat 8.0.32-- 41211  ---  
Tomcat 8.5.14  N/A  456001.67N/A
46000   1.67
-
Throughput: requests per sec.
ART: msecs

Also, forgot to add earlier, the jMeter client has keep-alive header enabled, 
and the server is left to use the default keepalive values.


-Original Message-
From: Piyush Kumar Nayak [mailto:pna...@adobe.com.INVALID] 
Sent: Tuesday, June 06, 2017 7:59 PM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: RE: Tomcat 8/NIO performance discrepancies

Violeta,

The data you see in an average of 3 samples of 180 second duration each.

Missed out on mentioning it before, but the thoughput is in no. of requests per 
second, and ART is in milliseconds.



Thanks

Piyush.



-Original Message-
From: Violeta Georgieva [mailto:violet...@apache.org]
Sent: Tuesday, June 06, 2017 7:09 PM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies



Hi,



2017-06-06 11:10 GMT+03:00 Piyush Kumar Nayak 
<pna...@adobe.com.invalid<mailto:pna...@adobe.com.invalid>>:

>

> Thanks, Mark.

>

> Here are some additional details.

>

> I am using Apache JMeter to inject load. I am using a simple

> hello-world

JSP.

>

> The test JMX simulates 100 concurrent user threads with a ramp-up of 5

secs and uses an HTTP Cookie Manager, that reuses cookies for each user thread.



Do you specify "Loop Count" for the number of iterations or you specify time 
duration?



Thanks,

Violeta



>

> I have disabled access log in tomcat. All the other server.xml

> settings

are the default.

>

>

> The BIO with Executor configuration we are using is:

>  maxThreads="150" minSpareThreads="4"/>

>

> connectionTimeout="2"

>redirectPort="8443" />

>

>

>

> It's the same for NIO. Just the protocol changes.

>

>

>

> The client(jMeter) and server (Tomcat) are on 2 separate physical

machines with the same configuration:

>

> RHEL 7.1 x64

>

> 16 core CPU;

>

> 32 GB RAM;

>

> 1Gbps NIC

>

>

>

> - JSP test

page 

>

> <%@ page language="java" contentType="text/html; charset=ISO-8859-1"

>

> pageEncoding="ISO-8859-1"%>

>

> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.w3.org%2FTR%2Fhtml4%2Floose.dtd=02%7C01%7C%7Ce541b882239d42e880f508d4ace1739f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636323531683671518=AwpErTK9ClLCwDZ23QnOqj195rlITTyAeG4JX0BFv0o%3D=0;>

>

> 

>

> 

>

> 

>

> Hello World - JSP tomcat test

>

> 

>

> 

>

> <%= "Hello World! - JSP on Tomcat 8.15" %>

>

> 

>

> 

>

> -  JSP

> test

page -

>

>

>

> Test result:

> helloWorld.JSP

>

>

>

> BIO

>

> NIO

>

> BIO w/ Exec

>

> NIO w/ Exec

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Tomcat 7.077

>

> 38600

>

> 2

>

> 42664

>

> 2

>

> 57104

>

> 1

>

> 43660

>

> 2

>

> Tomcat 8.0.44

>

> 38585

>

> 2

>

> 44752

>

> 2

>

> 63000

>

> 1

>

> 44341

>

> 2

>

> Tomcat 8.0.32

>

>

>

>

>

>

>

>

>

> 41211

>

>

&

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Violeta,

The data you see in an average of 3 samples of 180 second duration each.

Missed out on mentioning it before, but the thoughput is in no. of requests per 
second, and ART is in milliseconds.



Thanks

Piyush.



-Original Message-
From: Violeta Georgieva [mailto:violet...@apache.org]
Sent: Tuesday, June 06, 2017 7:09 PM
To: Tomcat Users List <users@tomcat.apache.org>
Subject: Re: Tomcat 8/NIO performance discrepancies



Hi,



2017-06-06 11:10 GMT+03:00 Piyush Kumar Nayak 
<pna...@adobe.com.invalid<mailto:pna...@adobe.com.invalid>>:

>

> Thanks, Mark.

>

> Here are some additional details.

>

> I am using Apache JMeter to inject load. I am using a simple

> hello-world

JSP.

>

> The test JMX simulates 100 concurrent user threads with a ramp-up of 5

secs and uses an HTTP Cookie Manager, that reuses cookies for each user thread.



Do you specify "Loop Count" for the number of iterations or you specify time 
duration?



Thanks,

Violeta



>

> I have disabled access log in tomcat. All the other server.xml

> settings

are the default.

>

>

> The BIO with Executor configuration we are using is:

>  maxThreads="150" minSpareThreads="4"/>

>

> connectionTimeout="2"

>redirectPort="8443" />

>

>

>

> It's the same for NIO. Just the protocol changes.

>

>

>

> The client(jMeter) and server (Tomcat) are on 2 separate physical

machines with the same configuration:

>

> RHEL 7.1 x64

>

> 16 core CPU;

>

> 32 GB RAM;

>

> 1Gbps NIC

>

>

>

> - JSP test

page 

>

> <%@ page language="java" contentType="text/html; charset=ISO-8859-1"

>

> pageEncoding="ISO-8859-1"%>

>

> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.w3.org%2FTR%2Fhtml4%2Floose.dtd=02%7C01%7C%7Ce541b882239d42e880f508d4ace1739f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636323531683671518=AwpErTK9ClLCwDZ23QnOqj195rlITTyAeG4JX0BFv0o%3D=0;>

>

> 

>

> 

>

> 

>

> Hello World - JSP tomcat test

>

> 

>

> 

>

> <%= "Hello World! - JSP on Tomcat 8.15" %>

>

> 

>

> 

>

> -  JSP

> test

page -

>

>

>

> Test result:

> helloWorld.JSP

>

>

>

> BIO

>

> NIO

>

> BIO w/ Exec

>

> NIO w/ Exec

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Throughput

>

> ART

>

> Tomcat 7.077

>

> 38600

>

> 2

>

> 42664

>

> 2

>

> 57104

>

> 1

>

> 43660

>

> 2

>

> Tomcat 8.0.44

>

> 38585

>

> 2

>

> 44752

>

> 2

>

> 63000

>

> 1

>

> 44341

>

> 2

>

> Tomcat 8.0.32

>

>

>

>

>

>

>

>

>

> 41211

>

>

>

>

>

>

>

> Tomcat 8.5.14

>

> N/A

>

> 45600

>

> 1.67

>

> N/A

>

> 46000

>

> 1.67

>

>

>

>

>

>

> Regards,

>

> Piyush.

>

>

>

> -Original Message-

>

> From: Mark Thomas [mailto:ma...@apache.org]

>

> Sent: Tuesday, June 06, 2017 12:50 PM

>

> To: Tomcat Users List 
> <users@tomcat.apache.org<mailto:users@tomcat.apache.org>>

>

> Subject: Re: Tomcat 8/NIO performance discrepancies

>

>

>

> On 06/06/17 07:30, Piyush Kumar Nayak wrote:

>

> > We have been testing different connectors of tomcat for performance.

>

> > We have tested the following server versions:

>

> >

>

> > -  Tomcat 7.077

>

> >

>

> > -  Tomcat 8.0.44

>

> >

>

> > -  Tomcat 8.0.32

>

> >

>

> > -  Tomcat 8.5.14

>

> >

>

> > with the following HTTP connectors:

>

> >

>

> > -  BIO

>

> >

>

> > -  BIO with Executor

>

> >

>

> > -  NIO

>

> >

>

> > -  NIO with Executor

>

> >

>

> > Our observation is that NIO is up to 15% faster than BIO. But BIO

> > with

Executor is up to 65% faster than BIO. Using a shared executor thread pool with 
NIO does not improve the performance. We have tried some variations of 
connector attribute with Tomcat 8 (m

Re: Tomcat 8/NIO performance discrepancies

[Violeta Georgieva]

Hi,

2017-06-06 11:10 GMT+03:00 Piyush Kumar Nayak <pna...@adobe.com.invalid>:
>
> Thanks, Mark.
>
> Here are some additional details.
>
> I am using Apache JMeter to inject load. I am using a simple hello-world
JSP.
>
> The test JMX simulates 100 concurrent user threads with a ramp-up of 5
secs and uses an HTTP Cookie Manager, that reuses cookies for each user
thread.

Do you specify "Loop Count" for the number of iterations or you specify
time duration?

Thanks,
Violeta

>
> I have disabled access log in tomcat. All the other server.xml settings
are the default.
>
>
> The BIO with Executor configuration we are using is:
>  maxThreads="150" minSpareThreads="4"/>
>
> connectionTimeout="2"
>redirectPort="8443" />
>
>
>
> It's the same for NIO. Just the protocol changes.
>
>
>
> The client(jMeter) and server (Tomcat) are on 2 separate physical
machines with the same configuration:
>
> RHEL 7.1 x64
>
> 16 core CPU;
>
> 32 GB RAM;
>
> 1Gbps NIC
>
>
>
> - JSP test
page 
>
> <%@ page language="java" contentType="text/html; charset=ISO-8859-1"
>
> pageEncoding="ISO-8859-1"%>
>
> http://www.w3.org/TR/html4/loose.dtd;>
>
> 
>
> 
>
> 
>
> Hello World - JSP tomcat test
>
> 
>
> 
>
> <%= "Hello World! - JSP on Tomcat 8.15" %>
>
> 
>
> 
>
> -  JSP test
page -
>
>
>
> Test result:
> helloWorld.JSP
>
>
>
> BIO
>
> NIO
>
> BIO w/ Exec
>
> NIO w/ Exec
>
> Throughput
>
> ART
>
> Throughput
>
> ART
>
> Throughput
>
> ART
>
> Throughput
>
> ART
>
> Tomcat 7.077
>
> 38600
>
> 2
>
> 42664
>
> 2
>
> 57104
>
> 1
>
> 43660
>
> 2
>
> Tomcat 8.0.44
>
> 38585
>
> 2
>
> 44752
>
> 2
>
> 63000
>
> 1
>
> 44341
>
> 2
>
> Tomcat 8.0.32
>
>
>
>
>
>
>
>
>
> 41211
>
>
>
>
>
>
>
> Tomcat 8.5.14
>
> N/A
>
> 45600
>
> 1.67
>
> N/A
>
> 46000
>
> 1.67
>
>
>
>
>
>
> Regards,
>
> Piyush.
>
>
>
> -Original Message-
>
> From: Mark Thomas [mailto:ma...@apache.org]
>
> Sent: Tuesday, June 06, 2017 12:50 PM
>
> To: Tomcat Users List <users@tomcat.apache.org>
>
> Subject: Re: Tomcat 8/NIO performance discrepancies
>
>
>
> On 06/06/17 07:30, Piyush Kumar Nayak wrote:
>
> > We have been testing different connectors of tomcat for performance.
>
> > We have tested the following server versions:
>
> >
>
> > -  Tomcat 7.077
>
> >
>
> > -  Tomcat 8.0.44
>
> >
>
> > -  Tomcat 8.0.32
>
> >
>
> > -  Tomcat 8.5.14
>
> >
>
> > with the following HTTP connectors:
>
> >
>
> > -  BIO
>
> >
>
> > -  BIO with Executor
>
> >
>
> > -  NIO
>
> >
>
> > -  NIO with Executor
>
> >
>
> > Our observation is that NIO is up to 15% faster than BIO. But BIO with
Executor is up to 65% faster than BIO. Using a shared executor thread pool
with NIO does not improve the performance. We have tried some variations of
connector attribute with Tomcat 8 (maxThreads, acceptorThreadCount), but
nothing brings us close to the performance that we can get with BIO with
Executor.
>
> >
>
> > We have upgraded to Tomcat 8.5, and can't seem to get it to perform as
well as Tomcat 7 using BIO with Executor, which is what we were using
earlier.
>
> >
>
> > I would appreciate any help with the following:
>
> >
>
> > -  Why is the executor not boosting the performance for NIO, as
it is for BIO?
>
>
>
> Without knowing the details of your tests - no idea. The numbers are a
long way from what I'd expect.
>
>
>
> >
>
> > -  BIO is not available as an option with Tomcat 8.5. Why has
it been removed? Is there any way to get it back ?
>
>
>
> Because it can't support the non-blocking requirements of the WebSocket
API or the Servlet API and while you can fake non-blocking support, the
result is liable to enter a deadlock.
>
>
>
> > I'd be happy to share any other details from my testing for any
clarification.
>
>
>
> You'll need to explain your testing methodology - in detail - before
anyone can provide any useful input.
>
>
>
> Mark
>
>
>
> -
>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

RE: Tomcat 8/NIO performance discrepancies

[Piyush Kumar Nayak]

Thanks, Mark.

Here are some additional details.

I am using Apache JMeter to inject load. I am using a simple hello-world JSP.

The test JMX simulates 100 concurrent user threads with a ramp-up of 5 secs and 
uses an HTTP Cookie Manager, that reuses cookies for each user thread.

I have disabled access log in tomcat. All the other server.xml settings are the 
default.


The BIO with Executor configuration we are using is:






It's the same for NIO. Just the protocol changes.



The client(jMeter) and server (Tomcat) are on 2 separate physical machines with 
the same configuration:

RHEL 7.1 x64

16 core CPU;

32 GB RAM;

1Gbps NIC



- JSP test page 


<%@ page language="java" contentType="text/html; charset=ISO-8859-1"

pageEncoding="ISO-8859-1"%>

http://www.w3.org/TR/html4/loose.dtd;>







Hello World - JSP tomcat test





<%= "Hello World! - JSP on Tomcat 8.15" %>





-  JSP test page 
-



Test result:
helloWorld.JSP



BIO

NIO

BIO w/ Exec

NIO w/ Exec

Throughput

ART

Throughput

ART

Throughput

ART

Throughput

ART

Tomcat 7.077

38600

2

42664

2

57104

1

43660

2

Tomcat 8.0.44

38585

2

44752

2

63000

1

44341

2

Tomcat 8.0.32









41211







Tomcat 8.5.14

N/A

45600

1.67

N/A

46000

1.67






Regards,

Piyush.



-Original Message-

From: Mark Thomas [mailto:ma...@apache.org]

Sent: Tuesday, June 06, 2017 12:50 PM

To: Tomcat Users List <users@tomcat.apache.org>

Subject: Re: Tomcat 8/NIO performance discrepancies



On 06/06/17 07:30, Piyush Kumar Nayak wrote:

> We have been testing different connectors of tomcat for performance.

> We have tested the following server versions:

>

> -  Tomcat 7.077

>

> -  Tomcat 8.0.44

>

> -  Tomcat 8.0.32

>

> -  Tomcat 8.5.14

>

> with the following HTTP connectors:

>

> -  BIO

>

> -  BIO with Executor

>

> -  NIO

>

> -  NIO with Executor

>

> Our observation is that NIO is up to 15% faster than BIO. But BIO with 
> Executor is up to 65% faster than BIO. Using a shared executor thread pool 
> with NIO does not improve the performance. We have tried some variations of 
> connector attribute with Tomcat 8 (maxThreads, acceptorThreadCount), but 
> nothing brings us close to the performance that we can get with BIO with 
> Executor.

>

> We have upgraded to Tomcat 8.5, and can't seem to get it to perform as well 
> as Tomcat 7 using BIO with Executor, which is what we were using earlier.

>

> I would appreciate any help with the following:

>

> -  Why is the executor not boosting the performance for NIO, as it is 
> for BIO?



Without knowing the details of your tests - no idea. The numbers are a long way 
from what I'd expect.



>

> -  BIO is not available as an option with Tomcat 8.5. Why has it been 
> removed? Is there any way to get it back ?



Because it can't support the non-blocking requirements of the WebSocket API or 
the Servlet API and while you can fake non-blocking support, the result is 
liable to enter a deadlock.



> I'd be happy to share any other details from my testing for any clarification.



You'll need to explain your testing methodology - in detail - before anyone can 
provide any useful input.



Mark



-

To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

[Mark Thomas]

On 06/06/17 07:30, Piyush Kumar Nayak wrote:
> We have been testing different connectors of tomcat for performance.
> We have tested the following server versions:
> 
> -  Tomcat 7.077
> 
> -  Tomcat 8.0.44
> 
> -  Tomcat 8.0.32
> 
> -  Tomcat 8.5.14
> 
> with the following HTTP connectors:
> 
> -  BIO
> 
> -  BIO with Executor
> 
> -  NIO
> 
> -  NIO with Executor
> 
> Our observation is that NIO is up to 15% faster than BIO. But BIO with 
> Executor is up to 65% faster than BIO. Using a shared executor thread pool 
> with NIO does not improve the performance. We have tried some variations of 
> connector attribute with Tomcat 8 (maxThreads, acceptorThreadCount), but 
> nothing brings us close to the performance that we can get with BIO with 
> Executor.
> 
> We have upgraded to Tomcat 8.5, and can't seem to get it to perform as well 
> as Tomcat 7 using BIO with Executor, which is what we were using earlier.
> 
> I would appreciate any help with the following:
> 
> -  Why is the executor not boosting the performance for NIO, as it is 
> for BIO?

Without knowing the details of your tests - no idea. The numbers are a
long way from what I'd expect.

> 
> -  BIO is not available as an option with Tomcat 8.5. Why has it been 
> removed? Is there any way to get it back ?

Because it can't support the non-blocking requirements of the WebSocket
API or the Servlet API and while you can fake non-blocking support, the
result is liable to enter a deadlock.

> I'd be happy to share any other details from my testing for any clarification.

You'll need to explain your testing methodology - in detail - before
anyone can provide any useful input.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org