Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 8:38 PM, Rallavagu wrote:
> 
> 
> On 3/10/16 5:23 PM, Christopher Schultz wrote: Rallavagu,
> 
> On 3/10/16 8:10 PM, Rallavagu wrote:
 
 
 On 3/10/16 2:33 PM, Christopher Schultz wrote: Rallavagu,
 
 On 3/10/16 5:16 PM, Rallavagu wrote:
>>> On 3/10/16 2:09 PM, Christopher Schultz wrote:
>>> Rallavagu,
>>> 
>>> On 3/10/16 4:02 PM, Rallavagu wrote:
>> On 3/10/16 11:54 AM, Christopher Schultz wrote:
>>> Are you sure you have matched-up the correct
>>> thread within the JVM that is using all that
>>> CPU?
>> 
>>> How are you measuring the CPU usage?
>> 
>> It would the ID output from "top -H" mapping to
>> "Native ID" in thread dump.
>>> 
>>> My version of 'top' (Debian Linux) doesn't show thread
>>> ids. :(
>>> 
>>> I seem to recall having to do some backflips to
>>> convert native thread id to Java thread id. Can you
>>> explain what you've done to do that?
>>> 
 A typical top -H shows the following
>>> 
 top - 11:40:11 up 190 days,  1:24,  1 user,  load
 average: 5.74, 6.09, 5.78 Tasks: 759 total,   4
 running, 755 sleeping,   0 stopped,   0 zombie
 Cpu(s): 18.4%us,  1.6%sy, 0.0%ni, 79.5%id, 0.1%wa,
 0.0%hi,  0.5%si, 0.0%st Mem: 8057664k total,
 7895252k used,   162412k free,63312k buffers
 Swap:  2064380k total, 199452k used,  1864928k free,
 2125868k cached
>>> 
 PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM
 TIME+ COMMAND 15648 tomcat20   0 9649m 4.8g 4520
 R 87.3 62.6 7:24.24 java 21710 tomcat20   0 9649m
 4.8g 4520 R 79.8 62.6 5:44.99 java 21694 tomcat20
 0 9649m 4.8g 4520 S 74.3 62.6 5:39.40 java 7889
 tomcat20   0 9649m 4.8g 4520 S 29.7 62.6 4:24.44
 java 7878 tomcat20   0 9649m 4.8g 4520 S 27.8
 62.6 4:36.82 java 21701 tomcat20   0 9649m 4.8g
 4520 S 26.0 62.6 5:49.83 java
>>> 
 After taking thread dump, I used Threadlogic which
 will show Native-ID as column which corresponds to
 PID shown above.
>>> 
 https://java.net/projects/threadlogic
>>> 
 This way it helps to determine the thread that might 
 potentially causing high cpu.
 
 Okay. Are you serving a high rate of requests? It's possible
 that the thread is just doing a lot of (legitimate) work.
 
 The BIO connector is very basic: it uses blocking reads, and
 the thread dump you showed before showed it waiting on IO, so
 it should be completely idle, using no CPU time.
 
 It's *possible* that it's in a busy-wait state where it is 
 performing a very short IO-wait in a loop that it never
 exits. But since you haven't specified any weird timeouts,
 etc. on your connector, I'm skeptical as to that being the
 cause.
 
 This thread stays at high CPU usage for quite a while? And
 every thread dump you do has the same (or very similar) stack
 trace?
 
> The symptom is that app is always high on CPU hovering
> between 75 - 85 and so looked at the thread dumps. Each
> thread dump shows few high cpu threads and some of those
> are supposedly idle. After looking at a particular thread
> by Id what it was doing in the previous thread some of
> these were reading on socket. So, that might be somewhat
> related to what you said about I/O wait.
 
> Also, if BIO is basic, what are other options?
> 
> https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_Imp
le
>
> 
mentation
> 
> The NIO connector is more scalable, but the BIO should use the
> *least* resources when handling modest loads. I wasn't suggesting
> that BIO should be avoided due to its simplicity... quite the
> contrary, I was suggesting that the BIO connector *should* be
> well-behaved.
> 
>> Sounds good. So, the thread might be simply "winding down" from
>> socket read? If many of those threads behaving the same way
>> pushing CPU to high. Is there anything that I should be looking
>> into or configure perhaps to address this or update to 7.0.68?

I would first upgrade to 7.0.68 and see if the problem goes away. If
not, report back. It would be good to know what kind of load the
server is under -- mostly number of requests per second you are
actually processing, and how many request-processing threads exist for
the Connector (to get an average throughput per thread).

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbiJKoACgkQ9CaO5/Lv0PA8XQCdELZ8f+rCs+UV7W4sPA819FgV
uuMAnjQamWQ3MnBz2DdJuoL9+GyJC4F0
=x9ql
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: 

Re: Idle Thread high CPU

[Rallavagu]

On 3/10/16 5:23 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 8:10 PM, Rallavagu wrote:



On 3/10/16 2:33 PM, Christopher Schultz wrote: Rallavagu,

On 3/10/16 5:16 PM, Rallavagu wrote:

On 3/10/16 2:09 PM, Christopher Schultz wrote: Rallavagu,

On 3/10/16 4:02 PM, Rallavagu wrote:

On 3/10/16 11:54 AM, Christopher Schultz wrote:

Are you sure you have matched-up the correct thread
within the JVM that is using all that CPU?



How are you measuring the CPU usage?


It would the ID output from "top -H" mapping to "Native
ID" in thread dump.


My version of 'top' (Debian Linux) doesn't show thread ids.
:(

I seem to recall having to do some backflips to convert
native thread id to Java thread id. Can you explain what
you've done to do that?


A typical top -H shows the following



top - 11:40:11 up 190 days,  1:24,  1 user,  load average:
5.74, 6.09, 5.78 Tasks: 759 total,   4 running, 755
sleeping,   0 stopped,   0 zombie Cpu(s): 18.4%us,  1.6%sy,
0.0%ni, 79.5%id, 0.1%wa,  0.0%hi,  0.5%si, 0.0%st Mem:
8057664k total,  7895252k used,   162412k free,63312k
buffers Swap:  2064380k total, 199452k used,  1864928k
free,  2125868k cached



PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND 15648 tomcat20   0 9649m 4.8g 4520 R 87.3 62.6
7:24.24 java 21710 tomcat20   0 9649m 4.8g 4520 R 79.8
62.6 5:44.99 java 21694 tomcat20   0 9649m 4.8g 4520 S
74.3 62.6 5:39.40 java 7889 tomcat20   0 9649m 4.8g
4520 S 29.7 62.6 4:24.44 java 7878 tomcat20   0 9649m
4.8g 4520 S 27.8 62.6 4:36.82 java 21701 tomcat20   0
9649m 4.8g 4520 S 26.0 62.6 5:49.83 java



After taking thread dump, I used Threadlogic which will
show Native-ID as column which corresponds to PID shown
above.



https://java.net/projects/threadlogic



This way it helps to determine the thread that might
potentially causing high cpu.


Okay. Are you serving a high rate of requests? It's possible that
the thread is just doing a lot of (legitimate) work.

The BIO connector is very basic: it uses blocking reads, and the
thread dump you showed before showed it waiting on IO, so it should
be completely idle, using no CPU time.

It's *possible* that it's in a busy-wait state where it is
performing a very short IO-wait in a loop that it never exits. But
since you haven't specified any weird timeouts, etc. on your
connector, I'm skeptical as to that being the cause.

This thread stays at high CPU usage for quite a while? And every
thread dump you do has the same (or very similar) stack trace?


The symptom is that app is always high on CPU hovering between 75
- 85 and so looked at the thread dumps. Each thread dump shows
few high cpu threads and some of those are supposedly idle. After
looking at a particular thread by Id what it was doing in the
previous thread some of these were reading on socket. So, that
might be somewhat related to what you said about I/O wait.



Also, if BIO is basic, what are other options?


https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_Imple
mentation

The NIO connector is more scalable, but the BIO should use the *least*
resources when handling modest loads. I wasn't suggesting that BIO
should be avoided due to its simplicity... quite the contrary, I was
suggesting that the BIO connector *should* be well-behaved.

Sounds good. So, the thread might be simply "winding down" from socket 
read? If many of those threads behaving the same way pushing CPU to 
high. Is there anything that I should be looking into or configure 
perhaps to address this or update to 7.0.68?



- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbiHf8ACgkQ9CaO5/Lv0PAHWQCbBHR4dIjpfv+K0qnJil50Buyk
CIAAoKUByrZhri66vkU/OLdJ/01qMc2u
=+9Fl
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 8:10 PM, Rallavagu wrote:
> 
> 
> On 3/10/16 2:33 PM, Christopher Schultz wrote: Rallavagu,
> 
> On 3/10/16 5:16 PM, Rallavagu wrote:
 On 3/10/16 2:09 PM, Christopher Schultz wrote: Rallavagu,
 
 On 3/10/16 4:02 PM, Rallavagu wrote:
>>> On 3/10/16 11:54 AM, Christopher Schultz wrote:
 Are you sure you have matched-up the correct thread
 within the JVM that is using all that CPU?
>>> 
 How are you measuring the CPU usage?
>>> 
>>> It would the ID output from "top -H" mapping to "Native
>>> ID" in thread dump.
 
 My version of 'top' (Debian Linux) doesn't show thread ids.
 :(
 
 I seem to recall having to do some backflips to convert
 native thread id to Java thread id. Can you explain what
 you've done to do that?
 
> A typical top -H shows the following
 
> top - 11:40:11 up 190 days,  1:24,  1 user,  load average:
> 5.74, 6.09, 5.78 Tasks: 759 total,   4 running, 755
> sleeping,   0 stopped,   0 zombie Cpu(s): 18.4%us,  1.6%sy,
> 0.0%ni, 79.5%id, 0.1%wa,  0.0%hi,  0.5%si, 0.0%st Mem:
> 8057664k total,  7895252k used,   162412k free,63312k
> buffers Swap:  2064380k total, 199452k used,  1864928k
> free,  2125868k cached
 
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+ 
> COMMAND 15648 tomcat20   0 9649m 4.8g 4520 R 87.3 62.6 
> 7:24.24 java 21710 tomcat20   0 9649m 4.8g 4520 R 79.8
> 62.6 5:44.99 java 21694 tomcat20   0 9649m 4.8g 4520 S
> 74.3 62.6 5:39.40 java 7889 tomcat20   0 9649m 4.8g
> 4520 S 29.7 62.6 4:24.44 java 7878 tomcat20   0 9649m
> 4.8g 4520 S 27.8 62.6 4:36.82 java 21701 tomcat20   0
> 9649m 4.8g 4520 S 26.0 62.6 5:49.83 java
 
> After taking thread dump, I used Threadlogic which will
> show Native-ID as column which corresponds to PID shown
> above.
 
> https://java.net/projects/threadlogic
 
> This way it helps to determine the thread that might
> potentially causing high cpu.
> 
> Okay. Are you serving a high rate of requests? It's possible that
> the thread is just doing a lot of (legitimate) work.
> 
> The BIO connector is very basic: it uses blocking reads, and the 
> thread dump you showed before showed it waiting on IO, so it should
> be completely idle, using no CPU time.
> 
> It's *possible* that it's in a busy-wait state where it is
> performing a very short IO-wait in a loop that it never exits. But
> since you haven't specified any weird timeouts, etc. on your
> connector, I'm skeptical as to that being the cause.
> 
> This thread stays at high CPU usage for quite a while? And every 
> thread dump you do has the same (or very similar) stack trace?
> 
>> The symptom is that app is always high on CPU hovering between 75
>> - 85 and so looked at the thread dumps. Each thread dump shows
>> few high cpu threads and some of those are supposedly idle. After
>> looking at a particular thread by Id what it was doing in the
>> previous thread some of these were reading on socket. So, that
>> might be somewhat related to what you said about I/O wait.
> 
>> Also, if BIO is basic, what are other options?

https://tomcat.apache.org/tomcat-8.0-doc/config/http.html#Standard_Imple
mentation

The NIO connector is more scalable, but the BIO should use the *least*
resources when handling modest loads. I wasn't suggesting that BIO
should be avoided due to its simplicity... quite the contrary, I was
suggesting that the BIO connector *should* be well-behaved.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbiHf8ACgkQ9CaO5/Lv0PAHWQCbBHR4dIjpfv+K0qnJil50Buyk
CIAAoKUByrZhri66vkU/OLdJ/01qMc2u
=+9Fl
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 5:16 PM, Rallavagu wrote:
> On 3/10/16 2:09 PM, Christopher Schultz wrote: Rallavagu,
> 
> On 3/10/16 4:02 PM, Rallavagu wrote:
 On 3/10/16 11:54 AM, Christopher Schultz wrote:
> Are you sure you have matched-up the correct thread within
> the JVM that is using all that CPU?
 
> How are you measuring the CPU usage?
 
 It would the ID output from "top -H" mapping to "Native ID"
 in thread dump.
> 
> My version of 'top' (Debian Linux) doesn't show thread ids. :(
> 
> I seem to recall having to do some backflips to convert native
> thread id to Java thread id. Can you explain what you've done to do
> that?
> 
>> A typical top -H shows the following
> 
>> top - 11:40:11 up 190 days,  1:24,  1 user,  load average: 5.74,
>> 6.09, 5.78 Tasks: 759 total,   4 running, 755 sleeping,   0
>> stopped,   0 zombie Cpu(s): 18.4%us,  1.6%sy,  0.0%ni, 79.5%id,
>> 0.1%wa,  0.0%hi,  0.5%si, 0.0%st Mem:   8057664k total,  7895252k
>> used,   162412k free,63312k buffers Swap:  2064380k total,
>> 199452k used,  1864928k free,  2125868k cached
> 
>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
>> COMMAND 15648 tomcat20   0 9649m 4.8g 4520 R 87.3 62.6
>> 7:24.24 java 21710 tomcat20   0 9649m 4.8g 4520 R 79.8 62.6
>> 5:44.99 java 21694 tomcat20   0 9649m 4.8g 4520 S 74.3 62.6
>> 5:39.40 java 7889 tomcat20   0 9649m 4.8g 4520 S 29.7 62.6
>> 4:24.44 java 7878 tomcat20   0 9649m 4.8g 4520 S 27.8 62.6
>> 4:36.82 java 21701 tomcat20   0 9649m 4.8g 4520 S 26.0 62.6
>> 5:49.83 java
> 
>> After taking thread dump, I used Threadlogic which will show
>> Native-ID as column which corresponds to PID shown above.
> 
>> https://java.net/projects/threadlogic
> 
>> This way it helps to determine the thread that might potentially
>> causing high cpu.

Okay. Are you serving a high rate of requests? It's possible that the
thread is just doing a lot of (legitimate) work.

The BIO connector is very basic: it uses blocking reads, and the
thread dump you showed before showed it waiting on IO, so it should be
completely idle, using no CPU time.

It's *possible* that it's in a busy-wait state where it is performing
a very short IO-wait in a loop that it never exits. But since you
haven't specified any weird timeouts, etc. on your connector, I'm
skeptical as to that being the cause.

This thread stays at high CPU usage for quite a while? And every
thread dump you do has the same (or very similar) stack trace?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh9igACgkQ9CaO5/Lv0PBFXQCeKARui/P0Z2/1dSBlwDcc8Dct
jA8AoLu7+oaTE2D6SlzNcXZdmIxtXVCq
=NzQR
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chuck,

On 3/10/16 5:18 PM, Caldarale, Charles R wrote:
>> From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
>> Subject: Re: Idle Thread high CPU
> 
>> My version of 'top' (Debian Linux) doesn't show thread ids. :(
> 
> Can you try "top -H" (case sensitive option)?

I'm not seeing individual threads when I run with -H. I just see PID.
I can get top to show me the "nTH" field (thread count), and it's
non-zero, but it's not showing me the individual threads.

Kernel level is 2.6.32 SMP. Might have something to do with it.

On another 3.2.0 SMP kernel, I do get the thread-id as PID and pid as
TGID.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh9WYACgkQ9CaO5/Lv0PCjDACfSbpulgrLsZa1ddclt1R5AdjO
KCAAn0jGd3wzDD/NRRB169aKBmjzPVRc
=VzIE
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Idle Thread high CPU

[Caldarale, Charles R]

> From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
> Subject: Re: Idle Thread high CPU

> My version of 'top' (Debian Linux) doesn't show thread ids. :(

Can you try "top -H" (case sensitive option)?

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Rallavagu]

On 3/10/16 2:09 PM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 4:02 PM, Rallavagu wrote:

On 3/10/16 11:54 AM, Christopher Schultz wrote:

Are you sure you have matched-up the correct thread within the
JVM that is using all that CPU?



How are you measuring the CPU usage?


It would the ID output from "top -H" mapping to "Native ID" in
thread dump.


My version of 'top' (Debian Linux) doesn't show thread ids. :(

I seem to recall having to do some backflips to convert native thread
id to Java thread id. Can you explain what you've done to do that?


A typical top -H shows the following

top - 11:40:11 up 190 days,  1:24,  1 user,  load average: 5.74, 6.09, 5.78
Tasks: 759 total,   4 running, 755 sleeping,   0 stopped,   0 zombie
Cpu(s): 18.4%us,  1.6%sy,  0.0%ni, 79.5%id,  0.1%wa,  0.0%hi,  0.5%si, 
0.0%st

Mem:   8057664k total,  7895252k used,   162412k free,63312k buffers
Swap:  2064380k total,   199452k used,  1864928k free,  2125868k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
15648 tomcat20   0 9649m 4.8g 4520 R 87.3 62.6   7:24.24 java
21710 tomcat20   0 9649m 4.8g 4520 R 79.8 62.6   5:44.99 java
21694 tomcat20   0 9649m 4.8g 4520 S 74.3 62.6   5:39.40 java
 7889 tomcat20   0 9649m 4.8g 4520 S 29.7 62.6   4:24.44 java
 7878 tomcat20   0 9649m 4.8g 4520 S 27.8 62.6   4:36.82 java
21701 tomcat20   0 9649m 4.8g 4520 S 26.0 62.6   5:49.83 java

After taking thread dump, I used Threadlogic which will show Native-ID 
as column which corresponds to PID shown above.


https://java.net/projects/threadlogic

This way it helps to determine the thread that might potentially causing 
high cpu.





Can you post your  and/or  configuration?





Okay, so you are using the default (BIO) connector with no special
configuration. I see you are running Tomcat 7.0.47. Could you please
re-test with latest 7.0.68? Your version is 2.5 years old and there
may be a performance bug fixed.


Sure.



- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh8LQACgkQ9CaO5/Lv0PA89wCdHBDkMdKa/9JkNLGY80Ygrl0/
NpkAoL5kQ8cx/Qr9WcOpdHw3DxvurcrD
=Ckbq
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 4:02 PM, Rallavagu wrote:
> On 3/10/16 11:54 AM, Christopher Schultz wrote:
>> Are you sure you have matched-up the correct thread within the
>> JVM that is using all that CPU?
> 
>> How are you measuring the CPU usage?
> 
> It would the ID output from "top -H" mapping to "Native ID" in
> thread dump.

My version of 'top' (Debian Linux) doesn't show thread ids. :(

I seem to recall having to do some backflips to convert native thread
id to Java thread id. Can you explain what you've done to do that?

>> Can you post your  and/or  configuration?
> 
>  maxThreads="500" connectionTimeout="2" redirectPort="28443" />

Okay, so you are using the default (BIO) connector with no special
configuration. I see you are running Tomcat 7.0.47. Could you please
re-test with latest 7.0.68? Your version is 2.5 years old and there
may be a performance bug fixed.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh8LQACgkQ9CaO5/Lv0PA89wCdHBDkMdKa/9JkNLGY80Ygrl0/
NpkAoL5kQ8cx/Qr9WcOpdHw3DxvurcrD
=Ckbq
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Rallavagu]

On 3/10/16 1:02 PM, Konstantin Kolinko wrote:

2016-03-10 22:54 GMT+03:00 Christopher Schultz :

Rallavagu,

On 3/10/16 2:11 PM, Rallavagu wrote:

 From a thread dump and corresponding "top" output it is reported
that the following thread is consuming significant CPU (around
80%)

"http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000
nid=0x54ce waiting on condition [0x7f4b038f7000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00071846da50> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)



at

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)



at

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.

awaitNanos(AbstractQueuedSynchronizer.java:2082)


  at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java

:467)




at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)

at
org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav

a:1068)


  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j

ava:1130)


  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.

java:615)


  at java.lang.Thread.run(Thread.java:744)

Considering it is a thread from tomcat's thread pool and it
suggests that it is "java.lang.Thread.State: TIMED_WAITING
(parking)" wonder why this shows up as consuming high CPU. Any
clues would be highly appreciated. Thanks in advance.


Are you sure you have matched-up the correct thread within the JVM
that is using all that CPU?

How are you measuring the CPU usage?

Can you post your  and/or  configuration?



Also


Sorry for not providing this information upfront.

1. Tomcat version =?

7.0.47

2. Java version =?

Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

3. You need to take several (e.g. 3) thread dumps to see whether there
is some progress /change of state.


There is some progress. But, similar but different thread(s) show high 
cpu in some of those.




The last similar thread in search results for users@ list is in May
2015 "High cpu on Tomcat 8", a year ago. As there were no such threads
recently, I think that bug has already been fixed.


The only Tomcat code in that stacktrace is
org.apache.tomcat.util.threads.TaskQueue.

The connector type can be guessed as "bio" from the thread name, but
as a thread name can be configured on an Executor, this identification
is not reliable.

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Konstantin Kolinko]

2016-03-10 22:54 GMT+03:00 Christopher Schultz :
> Rallavagu,
>
> On 3/10/16 2:11 PM, Rallavagu wrote:
>> From a thread dump and corresponding "top" output it is reported
>> that the following thread is consuming significant CPU (around
>> 80%)
>>
>> "http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000
>> nid=0x54ce waiting on condition [0x7f4b038f7000]
>> java.lang.Thread.State: TIMED_WAITING (parking) at
>> sun.misc.Unsafe.park(Native Method) - parking to wait for
>> <0x00071846da50> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>
> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>>
>>
> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> awaitNanos(AbstractQueuedSynchronizer.java:2082)
>>
>>  at
>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java
> :467)
>>
>>
> at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
>> at
>> org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav
> a:1068)
>>
>>  at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1130)
>>
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:615)
>>
>>  at java.lang.Thread.run(Thread.java:744)
>>
>> Considering it is a thread from tomcat's thread pool and it
>> suggests that it is "java.lang.Thread.State: TIMED_WAITING
>> (parking)" wonder why this shows up as consuming high CPU. Any
>> clues would be highly appreciated. Thanks in advance.
>
> Are you sure you have matched-up the correct thread within the JVM
> that is using all that CPU?
>
> How are you measuring the CPU usage?
>
> Can you post your  and/or  configuration?


Also
1. Tomcat version =?
2. Java version =?
3. You need to take several (e.g. 3) thread dumps to see whether there
is some progress /change of state.

The last similar thread in search results for users@ list is in May
2015 "High cpu on Tomcat 8", a year ago. As there were no such threads
recently, I think that bug has already been fixed.


The only Tomcat code in that stacktrace is
org.apache.tomcat.util.threads.TaskQueue.

The connector type can be guessed as "bio" from the thread name, but
as a thread name can be configured on an Executor, this identification
is not reliable.

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Rallavagu]

On 3/10/16 11:54 AM, Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 2:11 PM, Rallavagu wrote:

 From a thread dump and corresponding "top" output it is reported
that the following thread is consuming significant CPU (around
80%)

"http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000
nid=0x54ce waiting on condition [0x7f4b038f7000]
java.lang.Thread.State: TIMED_WAITING (parking) at
sun.misc.Unsafe.park(Native Method) - parking to wait for
<0x00071846da50> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)



at

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)



at

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.

awaitNanos(AbstractQueuedSynchronizer.java:2082)


  at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java

:467)




at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)

at
org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav

a:1068)


  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j

ava:1130)


  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.

java:615)


  at java.lang.Thread.run(Thread.java:744)

Considering it is a thread from tomcat's thread pool and it
suggests that it is "java.lang.Thread.State: TIMED_WAITING
(parking)" wonder why this shows up as consuming high CPU. Any
clues would be highly appreciated. Thanks in advance.


Are you sure you have matched-up the correct thread within the JVM
that is using all that CPU?

How are you measuring the CPU usage?


It would the ID output from "top -H" mapping to "Native ID" in thread dump.



Can you post your  and/or  configuration?




Thanks



- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh0Q0ACgkQ9CaO5/Lv0PDDpACfbmueOC3FIVAImoIY6P0GYgla
iUUAnRimWHnwhvNR+q2KKyypxnFcoHBi
=o7Ac
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Idle Thread high CPU

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rallavagu,

On 3/10/16 2:11 PM, Rallavagu wrote:
> From a thread dump and corresponding "top" output it is reported
> that the following thread is consuming significant CPU (around
> 80%)
> 
> "http-bio-28080-exec-437" daemon prio=10 tid=0x7f4acc0de000 
> nid=0x54ce waiting on condition [0x7f4b038f7000] 
> java.lang.Thread.State: TIMED_WAITING (parking) at
> sun.misc.Unsafe.park(Native Method) - parking to wait for
> <0x00071846da50> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
> 
at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>
> 
at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
awaitNanos(AbstractQueuedSynchronizer.java:2082)
>
>  at 
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java
:467)
>
> 
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
> at
> org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) 
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.jav
a:1068)
>
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1130)
>
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:615)
>
>  at java.lang.Thread.run(Thread.java:744)
> 
> Considering it is a thread from tomcat's thread pool and it
> suggests that it is "java.lang.Thread.State: TIMED_WAITING
> (parking)" wonder why this shows up as consuming high CPU. Any
> clues would be highly appreciated. Thanks in advance.

Are you sure you have matched-up the correct thread within the JVM
that is using all that CPU?

How are you measuring the CPU usage?

Can you post your  and/or  configuration?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlbh0Q0ACgkQ9CaO5/Lv0PDDpACfbmueOC3FIVAImoIY6P0GYgla
iUUAnRimWHnwhvNR+q2KKyypxnFcoHBi
=o7Ac
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org