Re: JVM Crash in tcnative due to concurrency/timing in HTTP/2

2017-06-12 Thread Rémy Maucherat 
2017-06-12 17:07 GMT-05:00 Mark Thomas :

> On 09/06/17 16:02, Kreuser, Peter wrote:
> > Hi all,
> >
> > Sorry for the long text. I hope somebody can help me track down the
> problem I'm facing with Tomcat (8.5.15), tcnative (1.2.12), openssl
> (1.1.0e) and HTTP/2. JVM is zulu-8.21.0.1 (1.8.0_131-b11)
>
> No need to apologise. There was enough information in this report for me
> to be 95% sure of what was going on before I tested things locally.
>
> 
>
> > Would be nice if somebody here could tell me if that is a bug or if I
> have some misconfiguration.
>
> It is a bug.
>
> The short explanation is that with NIO2, the style of the API (async
> read/write with CompletionHandlers) means that a TLS connection can get
> closed down (by a CompletionHandler) before the access log entry is
> written. This leads to passing a NULL reference to some native code
> which - unsurprisingly - falls over (Tomcat is meant to make sure it
> doesn't pass NULL references).
>
> There is an easy fix but the side-effect is that sometimes the cipher
> suite name won't appear in the access log. Depending on timing on your
> system that could be very frequent to almost never. I'd roughly expect
> it to happen as frequently as the crashes.
>
> The harder fix is to note that we want the info and to ensure we cache
> it before the TLS connection is closed down. In theory, we could do this
> in the access log but it means parsing the request attribute config.
> Performance wise, there will be an impact on users since the lookup to
> populate the cache will occur during request processing rather than
> afterwards. However, overall throughput should be unaffected.
>
> An alternative fix is for the application to trigger the caching of the
> TLS info by requesting one of the TLD attributes.
>
> Switching to NIO should also fix this. Note, I would not expect NIO and
> NIO2 to be much different performance wise.
>
> Would the easy fix be sufficient?
>
> Are you sure it's not simply still this bug:
https://bz.apache.org/bugzilla/show_bug.cgi?id=60461 ?
I agree the fix is "hard" since, although there's no actual async close (I
don't see any completion handler closing things abruptly personally - on
read I failed to do anything async for now, and on write the completion
handler simply sets an error field to be processed synchronously later),
closing can occur after parsing any frame. IMO with HTTP/2 everything is
async to the application, so it's the same for all connectors.

Rémy

Re: JVM Crash in tcnative due to concurrency/timing in HTTP/2

2017-06-12 Thread Mark Thomas 
On 09/06/17 16:02, Kreuser, Peter wrote:
> Hi all,
> 
> Sorry for the long text. I hope somebody can help me track down the problem 
> I'm facing with Tomcat (8.5.15), tcnative (1.2.12), openssl (1.1.0e) and 
> HTTP/2. JVM is zulu-8.21.0.1 (1.8.0_131-b11)

No need to apologise. There was enough information in this report for me
to be 95% sure of what was going on before I tested things locally.



> Would be nice if somebody here could tell me if that is a bug or if I have 
> some misconfiguration.

It is a bug.

The short explanation is that with NIO2, the style of the API (async
read/write with CompletionHandlers) means that a TLS connection can get
closed down (by a CompletionHandler) before the access log entry is
written. This leads to passing a NULL reference to some native code
which - unsurprisingly - falls over (Tomcat is meant to make sure it
doesn't pass NULL references).

There is an easy fix but the side-effect is that sometimes the cipher
suite name won't appear in the access log. Depending on timing on your
system that could be very frequent to almost never. I'd roughly expect
it to happen as frequently as the crashes.

The harder fix is to note that we want the info and to ensure we cache
it before the TLS connection is closed down. In theory, we could do this
in the access log but it means parsing the request attribute config.
Performance wise, there will be an impact on users since the lookup to
populate the cache will occur during request processing rather than
afterwards. However, overall throughput should be unaffected.

An alternative fix is for the application to trigger the caching of the
TLS info by requesting one of the TLD attributes.

Switching to NIO should also fix this. Note, I would not expect NIO and
NIO2 to be much different performance wise.

Would the easy fix be sufficient?

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

2017-06-12 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256



On 6/12/17 3:43 PM, Piyush Kumar Nayak wrote:
> Sure, I'll check it out and get back. But the last time I tried, 
> JMeter gave me better numbers than ab. Don't want the client to be 
> the bottleneck.

+1

ab evidently doesn't "do" high-concurrency very well. Something about
a single global-lock. Mladen and Jean-Frederic have more information
about it than I do.

- -chris

> -Original Message- From: Mark Thomas
> [mailto:ma...@apache.org] Sent: Tuesday, June 13, 2017 1:05 AM To:
> Tomcat Users List  Subject: Re: Tomcat
> 8/NIO performance discrepancies
> 
> On 12/06/17 20:27, Piyush Kumar Nayak wrote:
>> Thanks, Mark. I have been testing with 100 concurrent users.
>> While the executor had the maxThreads set to 150, the connector
>> without the executor was left to the default value (which should
>> be 200). I did not have the disableKeepAlivePercentage attribute
>> set, which means that it should take to the default value of 75.
>> 
>> So, the number of threads in use before the keepalive is disabled
>> in case of non-executor connector should be 150. At 100 users,
>> the no. of clients is at 50% of the maxThread count. Unless I'm
>> missing something here, I don't see why keep-alive would be
>> disabled. I'd appreciate if you can give me some pointers on how
>> you can tell if requests are processed at the server end with
>> keep-alive.
> 
> I suggest you use ab rather than JMeter. I've seen all sorts of odd
> behaviour with JMeter when processing requests very quickly that I
> haven't seen with ab. Also, ab tells you how many of your requests
> used keep-alive.
> 
> Mark
> 
> 
>> 
>> I'll try setting disableKeepAlivePercentage to 100, and see if
>> that improves the performance.
>> 
>> -Original Message- From: Mark Thomas
>> [mailto:ma...@apache.org] Sent: Thursday, June 08, 2017 7:13 PM 
>> To: Tomcat Users List  Subject: Re:
>> Tomcat 8/NIO performance discrepancies
>> 
>> On 07/06/17 22:28, Mark Thomas wrote:
>> 
>> 
>> 
>>> I'm still seeing significant differences in BIO with and
>>> without an executor.
>>> 
>>> I'm still looking for the root cause. I have noticed that
>>> without an executor I see far fewer keep-alive requests than
>>> I'd expect. That could explain the difference, or some of it at
>>> least.
>> 
>> Further testing indicates that most, if not all, of the
>> performance difference is down to the fewer number of requests
>> processed using HTTP keep-alive. This is because BIO without an
>> executor is subject to disableKeepAlivePercentage whereas BIO
>> with an executor is not.
>> 
>> If the number of concurrent clients is kept below 75% of
>> maxThreads, performance remains very similar.
>> 
>> The problems you are likely to experience in the real world
>> without disableKeepAlivePercentage (i.e. if you use BIO with an
>> executor) are described in this thread:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomca
t
>>
>> 
.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122
>> aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C6
3
>>
>> 
6325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D&
>> reserved=0
>> 
>> The better tuning options are probably: - increase maxThreads to
>> max expected client concurrency + headroom or if your memory/CPU
>> won't permit that, as high as you can - increase
>> disableKeepAlivePercentage
>> 
>> I'll look at BIO vs NIO next.
>> 
>> Mark
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJZPvXxAAoJEBzwKT+lPKRYCb8QAL555dkQlFCPEXsjqtP7IGIA
7eQjX7gE1Lxk3ycCmTcndyixFWGoGdLwp41OPadVYF9TSe003fiHbptiSsYgCCEM
HEOMisKNkVZg22Wyenm0eJLPaALMOE7xU95rmPpacVxgslPWmMLSsNk9TC/c2xQ1
1o7iiS9/hEF6LsoT5oOLxe/ZGp8jxTHXBgahvqCierzN8T089G7GomVS1Q/FDD2t
OLOu/JRcTUKzxlISONbchtF1fSc9HenLmrZxqVw52iZGLGrbsIAg8D/x7kPhAxbk
HLHG9GwkHh7iOavW25UWFVVlmC58XioDJJVZWTGzEhfxAU/zoyUZbZ2CtqSDTdUG
rcsBjk33A1LYc3yioq8i55fq7d84cLdG2Xf//kUaBq+yP3tuNpWYk/WEOL/kuQFp

RE: Tomcat 8/NIO performance discrepancies

2017-06-12 Thread Piyush Kumar Nayak 
Sure, I'll check it out and get back. 
But the last time I tried, JMeter gave me better numbers than ab. Don't want 
the client to be the bottleneck.
 

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Tuesday, June 13, 2017 1:05 AM
To: Tomcat Users List 
Subject: Re: Tomcat 8/NIO performance discrepancies

On 12/06/17 20:27, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> I have been testing with 100 concurrent users. While the executor had the 
> maxThreads set to 150, the connector without the executor was left to the 
> default value (which should be 200). I did not have the 
> disableKeepAlivePercentage attribute set, which means that it should take to 
> the default value of 75.
> 
> So, the number of threads in use before the keepalive is disabled in case of 
> non-executor connector should be 150. At 100 users, the no. of clients is at 
> 50% of the maxThread count. Unless I'm missing something here, I don't see 
> why keep-alive would be disabled.
> I'd appreciate if you can give me some pointers on how you can tell if 
> requests are processed at the server end with keep-alive.

I suggest you use ab rather than JMeter. I've seen all sorts of odd behaviour 
with JMeter when processing requests very quickly that I haven't seen with ab. 
Also, ab tells you how many of your requests used keep-alive.

Mark


> 
> I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
> the performance.
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org]
> Sent: Thursday, June 08, 2017 7:13 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 8/NIO performance discrepancies
> 
> On 07/06/17 22:28, Mark Thomas wrote:
> 
> 
> 
>> I'm still seeing significant differences in BIO with and without an 
>> executor.
>>
>> I'm still looking for the root cause. I have noticed that without an 
>> executor I see far fewer keep-alive requests than I'd expect. That 
>> could explain the difference, or some of it at least.
> 
> Further testing indicates that most, if not all, of the performance 
> difference is down to the fewer number of requests processed using HTTP 
> keep-alive. This is because BIO without an executor is subject to 
> disableKeepAlivePercentage whereas BIO with an executor is not.
> 
> If the number of concurrent clients is kept below 75% of maxThreads, 
> performance remains very similar.
> 
> The problems you are likely to experience in the real world without 
> disableKeepAlivePercentage (i.e. if you use BIO with an executor) are 
> described in this thread:
> 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat
> .markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122
> aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63
> 6325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D&
> reserved=0
> 
> The better tuning options are probably:
> - increase maxThreads to max expected client concurrency + headroom
>   or if your memory/CPU won't permit that, as high as you can
> - increase disableKeepAlivePercentage
> 
> I'll look at BIO vs NIO next.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

2017-06-12 Thread Piyush Kumar Nayak 
Mark,

I apologize for not bringing this up before, but is that BIO vs NIO or BIO with 
executor vs NIO with executor?
Thanks again, for looking into this.

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Friday, June 09, 2017 12:30 AM
To: Tomcat Users List 
Subject: Re: Tomcat 8/NIO performance discrepancies

On 08/06/17 14:42, Mark Thomas wrote:
> I'll look at BIO vs NIO next.

At the sort of request rates and response sizes we are talking about, we are a 
long way from real-world applications and the differences you see between BIO 
and NIO are largely down to the differences between doing a blocking read for 
the next request (BIO) and polling (NIO).

What you will notice is that as the number of concurrent clients rises, BIO 
performance will drop considerably as soon as the disableKeepAlivePercentage is 
exceeded and will continue to fall as client concurrency rises. NIO, on the 
other hand, will retain broadly the same performance even when client 
concurrency >> threads.

I have spent some time looking at this in detail with a profiler but the timing 
differences we are talking about are so small that often the profiler overhead 
ends up dominating.

I think the general lesson here is that benchmarks like this can be interesting 
but the further removed they are from real-world examples, the less useful they 
are likely to be.

As you move towards more real-world examples, what you tend to find is that I/O 
and/or application logic become far more dominant.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8/NIO performance discrepancies

2017-06-12 Thread Mark Thomas 
On 12/06/17 20:27, Piyush Kumar Nayak wrote:
> Thanks, Mark.
> I have been testing with 100 concurrent users. While the executor had the 
> maxThreads set to 150, the connector without the executor was left to the 
> default value (which should be 200). I did not have the 
> disableKeepAlivePercentage attribute set, which means that it should take to 
> the default value of 75.
> 
> So, the number of threads in use before the keepalive is disabled in case of 
> non-executor connector should be 150. At 100 users, the no. of clients is at 
> 50% of the maxThread count. Unless I'm missing something here, I don't see 
> why keep-alive would be disabled.
> I'd appreciate if you can give me some pointers on how you can tell if 
> requests are processed at the server end with keep-alive.

I suggest you use ab rather than JMeter. I've seen all sorts of odd
behaviour with JMeter when processing requests very quickly that I
haven't seen with ab. Also, ab tells you how many of your requests used
keep-alive.

Mark


> 
> I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
> the performance.
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org] 
> Sent: Thursday, June 08, 2017 7:13 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 8/NIO performance discrepancies
> 
> On 07/06/17 22:28, Mark Thomas wrote:
> 
> 
> 
>> I'm still seeing significant differences in BIO with and without an 
>> executor.
>>
>> I'm still looking for the root cause. I have noticed that without an 
>> executor I see far fewer keep-alive requests than I'd expect. That 
>> could explain the difference, or some of it at least.
> 
> Further testing indicates that most, if not all, of the performance 
> difference is down to the fewer number of requests processed using HTTP 
> keep-alive. This is because BIO without an executor is subject to 
> disableKeepAlivePercentage whereas BIO with an executor is not.
> 
> If the number of concurrent clients is kept below 75% of maxThreads, 
> performance remains very similar.
> 
> The problems you are likely to experience in the real world without 
> disableKeepAlivePercentage (i.e. if you use BIO with an executor) are 
> described in this thread:
> 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D=0
> 
> The better tuning options are probably:
> - increase maxThreads to max expected client concurrency + headroom
>   or if your memory/CPU won't permit that, as high as you can
> - increase disableKeepAlivePercentage
> 
> I'll look at BIO vs NIO next.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 8/NIO performance discrepancies

2017-06-12 Thread Piyush Kumar Nayak 
Thanks, Mark.
I have been testing with 100 concurrent users. While the executor had the 
maxThreads set to 150, the connector without the executor was left to the 
default value (which should be 200). I did not have the 
disableKeepAlivePercentage attribute set, which means that it should take to 
the default value of 75.

So, the number of threads in use before the keepalive is disabled in case of 
non-executor connector should be 150. At 100 users, the no. of clients is at 
50% of the maxThread count. Unless I'm missing something here, I don't see why 
keep-alive would be disabled.
I'd appreciate if you can give me some pointers on how you can tell if requests 
are processed at the server end with keep-alive.

I'll try setting disableKeepAlivePercentage to 100, and see if that improves 
the performance.

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: Thursday, June 08, 2017 7:13 PM
To: Tomcat Users List 
Subject: Re: Tomcat 8/NIO performance discrepancies

On 07/06/17 22:28, Mark Thomas wrote:



> I'm still seeing significant differences in BIO with and without an 
> executor.
> 
> I'm still looking for the root cause. I have noticed that without an 
> executor I see far fewer keep-alive requests than I'd expect. That 
> could explain the difference, or some of it at least.

Further testing indicates that most, if not all, of the performance difference 
is down to the fewer number of requests processed using HTTP keep-alive. This 
is because BIO without an executor is subject to disableKeepAlivePercentage 
whereas BIO with an executor is not.

If the number of concurrent clients is kept below 75% of maxThreads, 
performance remains very similar.

The problems you are likely to experience in the real world without 
disableKeepAlivePercentage (i.e. if you use BIO with an executor) are described 
in this thread:

https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftomcat.markmail.org%2Fthread%2Fmcfdk2273v2fsh5k=02%7C01%7C%7C6816158122aa4914b20208d4ae744a1e%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636325261860429837=8t1j6lxgc3aTCDq45mXbH7LqedtIuRgPsrgqUKXyCrc%3D=0

The better tuning options are probably:
- increase maxThreads to max expected client concurrency + headroom
  or if your memory/CPU won't permit that, as high as you can
- increase disableKeepAlivePercentage

I'll look at BIO vs NIO next.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org