date:20180116

Re: Ajp Nio-thread stuck in loop and consuming a lot of cpu

2018-01-16 Thread Mark Thomas

On 16/01/18 09:21, Toom Andreas wrote:
> Hi Mark,
> 
> We pulled out a CPU Call Tree using JVisualVM instead of YourKit and I have 
> uploaded a screenshot of it here: https://imgur.com/a/mqYxn
> 
> There is not much extra information compared to the java thread dump in my 
> initial post but it highlights in which code branch/method call most of the 
> cpu time is spent. Perhaps it will give some additional clues ? Let me know 
> if you think that it is worthwhile to use YourKit instead of JVisualVM and I 
> will try to get that for you instead.

Thanks. This has given me a couple of ideas. I need to do some more
testing. I hope to be able to get this done in the next day or so.

Mark


> 
> Best regards,
> Andreas 
> 
> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org] 
> Sent: den 12 januari 2018 11:27
> To: Tomcat Users List 
> Subject: Re: Ajp Nio-thread stuck in loop and consuming a lot of cpu
> 
> On 12/01/18 06:58, Toom Andreas wrote:
>> Hi,
>>
>> We are having an issue with an application running Apache Tomcat 8.0.47 
>> using Oracle Jvm 1.8.0.151 on Linux (RHEL 7). The Tomcat process is 
>> consuming  cpu at a constant high level even though there is a low amount of 
>> incoming traffic. When inspecting the process health using JMX /JVisualVM 
>> CPU Sampler we see that there are 4 ajp-nio-exec threads together with a 
>> NioBlockingSelector.BlockPoller thread that consume most of the cpu.
> 
> Are you able to determine the code path of the loop? A single stack trace can 
> point us in the right direction but often the root cause can still be tricky 
> to track down.
> 
> A profiler such as YourKit in CPU tracing mode should be able to provide 
> enough info to figure this out (and won't require any code changes to your 
> system).
> 
> Remote debugging is the other option but that is less ideal in production.
> 
> Mark
> 
> 
>>
>> A stack trace of one of the ajp-io-exec threads looks like this:
>>
>> "ajp-nio-48317-exec-14233" - Thread t@201195
>>java.lang.Thread.State: TIMED_WAITING
>>  at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for <342fab60> (a 
>> java.util.concurrent.CountDownLatch$Sync)
>>  at 
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>  at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>>  at 
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>>  at 
>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
>>  at 
>> org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitLatch(NioEndpoint.java:1400)
>>  at 
>> org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitReadLatch(NioEndpoint.java:1402)
>>  at 
>> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
>>  at 
>> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:250)
>>  at 
>> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:231)
>>  at 
>> org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:194)
>>  at 
>> org.apache.coyote.ajp.AjpNioProcessor.read(AjpNioProcessor.java:160)
>>  at 
>> org.apache.coyote.ajp.AbstractAjpProcessor.readMessage(AbstractAjpProcessor.java:1091)
>>  at 
>> org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:804)
>>  at 
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
>>  at 
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
>>  at 
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
>>  - locked <279cc7f7> (a 
>> org.apache.tomcat.util.net.NioChannel)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>  at 
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>  at java.lang.Thread.run(Thread.java:748)
>>
>>Locked ownable synchronizers:
>>  - locked <10e1f474> (a 
>> java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>> A stack trace of the NioBlockingSelector.BlockPoller thread looks like this:

Re: AJP Connector not throwing EOFException

2018-01-16 Thread emah

Hi Chris, 

thank you for responding promptly. 

Let me first correct something I've said earlier: 

HTTP Connector will actually throw an IOException when ssl unwrap fails
(rather than an EOFException)
  at
org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:618)
  at
org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:173)
  at
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:235)
  at
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:216)
  at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1241)
  at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1190)
  at
org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:717)
  at
org.apache.coyote.http11.Http11InputBuffer.access$300(Http11InputBuffer.java:40)
  at
org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1072)
  at
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:140)
  at
org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:261)
  at org.apache.coyote.Request.doRead(Request.java:581)
  at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326)
  at
org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642)
  at
org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:337)
  at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:93)

AJP Connector will just return -1
  at org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:600)
  at
org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.java:695)
  at
org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProcessor.java:1433)
  at org.apache.coyote.Request.doRead(Request.java:581)
  at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326)
  at
org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642)
  at
org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:337)
  at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:93)

I guess my question is whether I can make the AJP Connector provide a
similar hint (i.e. some sort of IOException) when connection to the
intermediary apache server is terminated.

Thanks, 
George



--
Sent from: http://tomcat.10.x6.nabble.com/Tomcat-User-f1968778.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AJP Connector not throwing EOFException

2018-01-16 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Emanuel,

On 1/16/18 10:13 AM, Emanuel Hategan wrote:
> I'm running a tomcat 8.5.23 instance on ubuntu 16.04 (spring boot 
> application with embedded tomcat) configured with 2 connectors: 
> Http11NioProtocol and AjpNioProtocol. The AJP one is accessed
> through an apache2 instance configured with mod_jk.It all works
> well in the normal use case.
> 
> The application has code to look for the EOFExceptions during read
> e.g. the client aborts the request which works well with HTTP but
> not AJP. In my test I'm simulating this by closing the connection
> half way through (some headers have been sent but no body)
> 
> The problem I'm seeing is an inconsistent behaviour for 
> CoyoteInputStream.read() The HTTP connector throws an EOFException
> in this case while the AJP one just returns -1.
> 
> The relevant call stack is this at 
> org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.java:
684)
>
> 
at
> org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProcess
or.java:1433)
>
> 
at org.apache.coyote.Request.doRead(Request.java:581)
> at 
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.ja
va:326)
>
> 
at
> org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuff
er.java:642)
>
> 
at
> org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:33
7)
>
> 
at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream
.java:93)
>
>  Now based on this old and unrelated thread 
> https://mail-archives.apache.org/mod_mbox/tomcat-users/201312.mbox/%3C
15ff6f04-b4c9-4d9b-b1b3-5c10ca955...@360works.com%3E
>
> 
I understand that the AJP connector is perfectly capable of raising an
> EOFException but it's just not doing that for me.
> 
> The documentation
> https://tomcat.apache.org/tomcat-8.0-doc/config/ajp.html does not
> suggest I need to do anything special.
> 
> I guess it's possible that this has something to do with the way
> apache2 talks to the AJP connector. Any help is appreciated.

Interesting.

If your servlet simply reads the InputStream like this, you don't get
an EOFException?

ServletInputStream in = request.getInputStream();
for(;;)
  in.read();

What part of the Servlet specification or Servlet API leads you to
believe that EOFException should be thrown when the request has been
completely read (or has been truncated, and no further data is available
)?

I don't see anything to suggest that such behavior is either required
or expected.

In fact, I'm surprised that the HTTP connector throws an EOFException
instead of simply returning -1 like the API says it should.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpeHIodHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjbTxAAwai0iMmly2VgCpJN
2JwON09y1X42+5G+xf/EgVf8CXlCVLUUlJKhbnwG9bJ8sJU0gi2/Uepk+oaOq7PV
pPNNA1PJBkKIQf3y3LPEbA5qIn5g/ucq1J5bgP5fCLAw6tmlNbHTwChQwhXKMOl0
UH+ehSo2BMaO1wXalNBY1+W60ZkaXSkvlXhVOICP8cDE3MWftGh6vLxjRtPLHVvK
219+xdRF7Bp7LnMdfg6PvI0eFpODGHMmAAShAaxZkNLR9ltebOkjY1e/HPgfP8iK
26Svf32nJNciKVOTJUW6W8JhiJpxz82sbiCotnNVMPBt2NoJbOjBjESDQgY7kjqm
/oUoK8AE4a30BUv40ZPzGhBsoFhHCHkEeilNwVhFtK76wmyyqh364s4cRTCnAWke
QJ6BGFXTXENbOIhgJeCJBi4IkliXKlIoxJdSpqnnA6osdb/5sjgEmflLoZavMz22
rV1CdRM7JTRd/hw9DjqUAFm87xtz/eGe37B67UIUhLes/c8VTIg+TlU65R0COTu8
PwY4U6SOXu/hcgdmSJxZ+PVREC6t5hiR8stNyulCGED1W2t52LXdr/q442ihZhpV
oEUwcDbAx3Ue6qDJq8tVpU1k7D8/Bn3smIuKdIoKXHIT4olSmDM4xh73ZwaISBzv
5bWCtDWNX2jgb5jK2K352m19364=
=udZE
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AJP Connector not throwing EOFException

2018-01-16 Thread Emanuel Hategan

I'm running a tomcat 8.5.23 instance on ubuntu 16.04 (spring boot
application with embedded tomcat) configured with 2 connectors:
Http11NioProtocol and AjpNioProtocol.
The AJP one is accessed through an apache2 instance configured with
mod_jk.It all works well in the normal use case.

The application has code to look for the EOFExceptions during read e.g. the
client aborts the request which works well with HTTP but not AJP.
In my test I'm simulating this by closing the connection half way through
(some headers have been sent but no body)

The problem I'm seeing is an inconsistent behaviour for
CoyoteInputStream.read()
The HTTP connector throws an EOFException in this case while the AJP one
just returns -1.

The relevant call stack is this
  at
org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.java:684)
  at
org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProcessor.java:1433)
  at org.apache.coyote.Request.doRead(Request.java:581)
  at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326)
  at
org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642)
  at
org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:337)
  at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:93)

Now based on this old and unrelated thread
https://mail-archives.apache.org/mod_mbox/tomcat-users/201312.mbox/%3c15ff6f04-b4c9-4d9b-b1b3-5c10ca955...@360works.com%3E
I understand that the AJP connector is perfectly capable of raising an
EOFException but it's just not doing that for me.

The documentation https://tomcat.apache.org/tomcat-8.0-doc/config/ajp.html
does not suggest I need to do anything special.

I guess it's possible that this has something to do with the way apache2
talks to the AJP connector. Any help is appreciated.

Regards,
George

RE: Ajp Nio-thread stuck in loop and consuming a lot of cpu

2018-01-16 Thread Toom Andreas

Hi Mark,

We pulled out a CPU Call Tree using JVisualVM instead of YourKit and I have 
uploaded a screenshot of it here: https://imgur.com/a/mqYxn

There is not much extra information compared to the java thread dump in my 
initial post but it highlights in which code branch/method call most of the cpu 
time is spent. Perhaps it will give some additional clues ? Let me know if you 
think that it is worthwhile to use YourKit instead of JVisualVM and I will try 
to get that for you instead.

Best regards,
Andreas 

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org] 
Sent: den 12 januari 2018 11:27
To: Tomcat Users List 
Subject: Re: Ajp Nio-thread stuck in loop and consuming a lot of cpu

On 12/01/18 06:58, Toom Andreas wrote:
> Hi,
> 
> We are having an issue with an application running Apache Tomcat 8.0.47 using 
> Oracle Jvm 1.8.0.151 on Linux (RHEL 7). The Tomcat process is consuming  cpu 
> at a constant high level even though there is a low amount of incoming 
> traffic. When inspecting the process health using JMX /JVisualVM CPU Sampler 
> we see that there are 4 ajp-nio-exec threads together with a 
> NioBlockingSelector.BlockPoller thread that consume most of the cpu.

Are you able to determine the code path of the loop? A single stack trace can 
point us in the right direction but often the root cause can still be tricky to 
track down.

A profiler such as YourKit in CPU tracing mode should be able to provide enough 
info to figure this out (and won't require any code changes to your system).

Remote debugging is the other option but that is less ideal in production.

Mark


> 
> A stack trace of one of the ajp-io-exec threads looks like this:
> 
> "ajp-nio-48317-exec-14233" - Thread t@201195
>java.lang.Thread.State: TIMED_WAITING
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for <342fab60> (a 
> java.util.concurrent.CountDownLatch$Sync)
>  at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
>  at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>  at 
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
>  at 
> org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitLatch(NioEndpoint.java:1400)
>  at 
> org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitReadLatch(NioEndpoint.java:1402)
>  at 
> org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
>  at 
> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:250)
>  at 
> org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:231)
>  at 
> org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:194)
>  at 
> org.apache.coyote.ajp.AjpNioProcessor.read(AjpNioProcessor.java:160)
>  at 
> org.apache.coyote.ajp.AbstractAjpProcessor.readMessage(AbstractAjpProcessor.java:1091)
>  at 
> org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:804)
>  at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
>  at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
>  at 
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
>  - locked <279cc7f7> (a 
> org.apache.tomcat.util.net.NioChannel)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>  at java.lang.Thread.run(Thread.java:748)
> 
>Locked ownable synchronizers:
>  - locked <10e1f474> (a 
> java.util.concurrent.ThreadPoolExecutor$Worker)
> 
> A stack trace of the NioBlockingSelector.BlockPoller thread looks like this:
> 
> "NioBlockingSelector.BlockPoller-2" - Thread t@17
>java.lang.Thread.State: RUNNABLE
>  at sun.nio.ch.EPollArrayWrapper.epollWait(Native 
> Method)
>  at 
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>

RE: Activating Tomcat 8.5 APR on RHEL7

2018-01-16 Thread Jean Pierre Urkens

Thanks, this worked.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Ajp Nio-thread stuck in loop and consuming a lot of cpu

Re: AJP Connector not throwing EOFException

Re: AJP Connector not throwing EOFException

AJP Connector not throwing EOFException

RE: Ajp Nio-thread stuck in loop and consuming a lot of cpu

RE: Activating Tomcat 8.5 APR on RHEL7

6 matches

Site Navigation

Mail list logo

Footer information