Ajp Nio-thread stuck in loop and consuming a lot of cpu

[Toom Andreas]

Hi,

We are having an issue with an application running Apache Tomcat 8.0.47 using 
Oracle Jvm 1.8.0.151 on Linux (RHEL 7). The Tomcat process is consuming  cpu at 
a constant high level even though there is a low amount of incoming traffic. 
When inspecting the process health using JMX /JVisualVM CPU Sampler we see that 
there are 4 ajp-nio-exec threads together with a 
NioBlockingSelector.BlockPoller thread that consume most of the cpu.

A stack trace of one of the ajp-io-exec threads looks like this:

"ajp-nio-48317-exec-14233" - Thread t@201195
   java.lang.Thread.State: TIMED_WAITING
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <342fab60> (a 
java.util.concurrent.CountDownLatch$Sync)
 at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
 at 
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
 at 
org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitLatch(NioEndpoint.java:1400)
 at 
org.apache.tomcat.util.net.NioEndpoint$KeyAttachment.awaitReadLatch(NioEndpoint.java:1402)
 at 
org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:185)
 at 
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:250)
 at 
org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:231)
 at 
org.apache.coyote.ajp.AjpNioProcessor.readSocket(AjpNioProcessor.java:194)
 at 
org.apache.coyote.ajp.AjpNioProcessor.read(AjpNioProcessor.java:160)
 at 
org.apache.coyote.ajp.AbstractAjpProcessor.readMessage(AbstractAjpProcessor.java:1091)
 at 
org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:804)
 at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
 at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
 at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
 - locked <279cc7f7> (a 
org.apache.tomcat.util.net.NioChannel)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
 - locked <10e1f474> (a 
java.util.concurrent.ThreadPoolExecutor$Worker)

A stack trace of the NioBlockingSelector.BlockPoller thread looks like this:

"NioBlockingSelector.BlockPoller-2" - Thread t@17
   java.lang.Thread.State: RUNNABLE
 at sun.nio.ch.EPollArrayWrapper.epollWait(Native 
Method)
 at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
 at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
 at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 - locked <648895e4> (a sun.nio.ch.Util$3)
 - locked <17921532> (a 
java.util.Collections$UnmodifiableSet)
 - locked <67762111> (a 
sun.nio.ch.EPollSelectorImpl)
 at 
sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
 at 
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:339)

   Locked ownable synchronizers:
 - None

It seems like the worker thread is trying to read from the underlying socket. 
When we inspect the OS tcp/ip statistics using the ss-command (ss -i src 
:)  the following connections are suspicious:

State  Recv-Q Send-Q Local Address:Port Peer Address:Port
CLOSE-WAIT 2  0  10.x.x.x:4831710.x.x.11:53724
  cubic wscale:7,7 rto:202 rtt:1.724/2.424 ato:40 
mss:1448 cwnd:10 ssthresh:31 bytes_acked:343869507 bytes_received:22259372 
segs_out:302687 segs_in:141943 send 67.2Mbps lastsnd:1290791171 
lastrcv:1290786778 lastack:1290786767 pacing_rate 134.4Mbps 

Re: Monitoring Tomcat JDBC Connection Pool with JConsole?

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jerry,

Please don't hijack threads. Start a new thread by sending a fresh
message to users@tomcat.apache.org. Don't just reply to an existing
message in the list and change the subject.

- -chris

On 1/11/18 1:47 PM, Jerry Malcolm wrote:
> I followed the instructions to enable JMX on Tomcat.  I added the 
> following lines to java config:
> 
> -Dcom.sun.management.jmxremote 
> -Dcom.sun.management.jmxremote.port=8083 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Djava.rmi.server.hostname=localhost
> 
> I opened JConsole localhost;8083, and it connected.   I can see
> CPU, heap, classes, and thread graphs.  When I look at the MBean
> tab, I see a Catalina entry and a "DataSource" entry below that.
> Expanding that gives me a ton of attributes, operations, and
> notifications for my various datasources.  That's fine.  But what I
> want is to be able to do is monitor a graph of the connection
> pools.  I saw an example graph on a 3rd party web post from several
> years ago.  But I can't find anything that shows connection pool
> usage graphs in my JConsole.
> 
> What am I missing?
> 
> Thanks.
> 
> Jerry
> 
> 
> -
>
> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpX558dHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFggTw/8DLOx0f30zEwSP4an
n6yAq6MaPQhOCLvGHSGRwspfM9ehsrSduAODO6BMx7/lZt+YTIm5/b8YRByN/UKT
gmUX/bcqpAyz9ZcKefnKqhfNLhBLD5QkRM1xqavfV8gvWuzVrVXRuP0rYKqP62pz
GKsyni6dj318cYq2d2QpdNyWjAMEnzbuOUsfGO5D34Ua/u8v/sHXtMPKBQMjr68h
pBal7OSbvo7tTxWUYQ3PutsnAlanE/lf9zh5bSuFPvFVVR7tPme5AltHt0CU3mlq
Q172uW+DN0faTvWpg0ha+dL5KgAdWRjagmcD30jX5tlTMGJAlbp2ItrCV/1Mdrq9
euAI6sMUdsxREXFfoljEusoE80Wx4HYqtivKQ3evkaVdO/Oq32ZyDQG6b2MiScsP
FvJ/vk8Vc6yyhXCOVb5NfyKRt9pvRyWotzyYzKL0X8GUYB9Asjnlwa1Cj/fRdMDe
DITKO8RIOyvGBXxXzvnHPYIJdCKc1Vsby/5GmiklEigxpcHZZ7mdX7kPnlfmlmdB
C5FrtNtO40Oe3Es5bi0Q17QD4HPw6NIYFwKzlfka6p4XtXW6ejTud6wyfHB2S6KQ
uwA0xHrIttSmh1Sj/dBETF8tcf9N1OyAOdSwoSPaL0G0gDT6n9kEhcNVZGa7uixF
/SXO8YiaWfdPXJv/elF8Dof2Ucs=
=VxSx
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Jasper Exception when using Tomcat 9 but works with Tomcat 5.5

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Sean,

On 1/11/18 11:58 AM, Sean Brett wrote:
> On 11/01/2018 15:48, "Christopher Schultz"
>  wrote:
> 
> 
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
>> 
>> Sean,
>> 
>> (Thanks for moving to the mailing list; it's a much better forum
>> for getting into a protracted discussion. See below for more,
>> inline.)
>> 
>> On 1/10/18 9:08 AM, Sean Brett wrote:
>>> I've been tasked with migrating a site from one institution to 
>>> another. As part of the process we are attempting to update
>>> the versions the site uses.
>>> 
>>> Initially I was asked to use Java 8 with Tomcat 5.5 (on
>>> Linux), which led to issues - not to mention Tomcat 5 being
>>> 'out of support¹.  I have since tried to deploy the webapps to
>>> Tomcat 9 with Java 8. Some of the webapps appear to run fine
>>> (and resolve the issues I was having with Tomcat5). However,
>>> the main app (and 2 similar/smaller web apps) throws a
>>> JasperException when I go to the index.jsp - the same web apps
>>> deployed to Tomcat 5.5 load without issue.
>> 
>> Moving to a later Tomcat version is best. We'll help you figure
>> out these issues so you can use a current version and be
>> better-equipped to stay current into the future.
>> 
>>> Most of the similar issues I have found have suggested this
>>> would be related to the jar files and possibly some sort of
>>> conflict. However, the lib structure for tomcat 9 is different
>>> to tomcat5 and I'm not sure which jars could be causing this.
>>> e.g. 
>>> https://stackoverflow.com/questions/22552244/tomcat-7-fails-to-compi
le
>>
>>> 
- - -jsp-
>>> 
>>> 
>> pages
>> 
>> JAR files might conflict, but that depends upon what you have
>> done so far. Easy questions:
>> 
>> 1. Have you added anything to Tomcat's lib/ directory (that would
>> be CATALINA_HOME/lib or CATALINA_BASE/lib -- or both) after
>> installing Tomcat 8/9?
> 
> Yes, I’ve added some jars.  Some I can’t remember why I’ve added
> them but doubt they’d be connected to the problem.  Here’s a list
> of the CATALINA_HOME/lib directory: (the last 4 being the ones that
> have been added)
> 
> [snip]
> 
> -rw-r--r--. 1 tomcat tomcat  313898 Dec 20 13:57 dom4j-1.6.1.jar 
> -rw-r--r--. 1 tomcat tomcat  802494 Dec 20 13:57
> freemarker-2.3.8.jar -rw-r--r--. 1 tomcat tomcat  355030 Dec 20
> 13:57 mail.jar -rw-r--r--. 1 tomcat tomcat  713037 Dec 20 14:15
> postgresql-42.1.4.jar

Yup, those look "foreign".

If possible, I'd move all of those except for the postgresql driver
into your web application's WEB-INF/lib folder. We can move them back
if absolutely necessary.

Is "mail.jar" the JavaMail API/implementation? You'd only need that in
Tomcat's lib/ directory if you were asking Tomcat to provide mail
session via JNDI -- such would appear somewhere in either your web
applications' META-INF/context.xml file or some file under
CATALINA_BASE/conf/*

Check your version of JavaMail. There are older versions which contain
vulnerabilities that have been patched. The API is (AFAICT) 100%
backward-compatible, so you should be able to upgrade to
JavaMail.latest without any problems. YMMV.

>> 2. What JAR files are bundled with your application in
>> WEB-INF/lib ?
> 
> -rw-r-. 1 root root   97703 Jan 11 15:05 servlet-api.jar

Remove that one. Tomcat should refuse to load it, but you should
remove it anyway.

> -rw-r-. 1 root root 8356134 Jan 11 15:05 rt.jar

Is that the Java JDK's runtime library? If so, remove that as well.
Tomcat should refuse to load classes over-top of the standard java.*,
javax.*, and a few other top-level packages but it's always possible
something might sneak in there and confuse everything.

> -rw-r-. 1 root root  474746 Jan 11 15:05
> postgresql-8.3-603.jdbc4.jar> -rw-r-. 1 root root  355030 Jan
> 11 15:05 mail.jar

If you have either (or both) of these in Tomcat's CATALINA_BASE/lib or
CATALINA_HOME/lib directories, you'll want to remove them from here.

> -rw-r-. 1 root root  352668 Jan 11 15:05 log4j-1.2.8.jar

Ok.

> -rw-r-. 1 root root   55008 Jan 11 15:05 local-ldap-0.0.1.jar 
> -rw-r-. 1 root root   38114 Jan 11 15:05 ldap-src-0.0.1.jar 
> -rw-r-. 1 root root   38668 Jan 11 15:05 ldap.jar -rw-r-. 1
> root root   55827 Jan 11 15:05 ldap-0.0.1.jar

Hmm. Does your application use LDAP directly?

> -rw-r-. 1 root root  146718 Jan 11 15:05 jdom.jar -rw-r-. 1
> root root  173211 Jan 11 15:05 IonoMap.jar -rw-r-. 1 root root
> 457002 Jan 11 15:05 ibmjndi.jar -rw-r-. 1 root root   38015 Jan
> 11 15:05 commons-logging-1.0.4.jar -rw-r-. 1 root root   87776
> Jan 11 15:05 commons-io-1.3.2.jar -rw-r-. 1 root root   22379
> Jan 11 15:05 commons-fileupload.jar -rw-r-. 1 root root   57779
> Jan 11 15:05 commons-fileupload-1.2.1.jar -rw-r-. 1 root root
> 559366 Jan 11 15:05 commons-collections-3.1.jar -rw-r-. 1 root
> root   58735 Jan 11 15:05 common-piims-src.jar -rw-r-. 1 root
> root  

Re: Monitoring Tomcat JDBC Connection Pool with JConsole?

[Simon De Uvarow]

"No olvides, no traiciones, lo que llevas bien dentro de ti. No olvides, no
traiciones, lo que siempre te ha hecho vivir."

On Thu, Jan 11, 2018 at 6:47 PM, Jerry Malcolm 
wrote:

> I followed the instructions to enable JMX on Tomcat.  I added the
> following lines to java config:
>
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=8083
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Djava.rmi.server.hostname=localhost
>
> I opened JConsole localhost;8083, and it connected.   I can see CPU, heap,
> classes, and thread graphs.  When I look at the MBean tab, I see a Catalina
> entry and a "DataSource" entry below that.  Expanding that gives me a ton
> of attributes, operations, and notifications for my various datasources.
> That's fine.  But what I want is to be able to do is monitor a graph of the
> connection pools.  I saw an example graph on a 3rd party web post from
> several years ago.  But I can't find anything that shows connection pool
> usage graphs in my JConsole.
>
> What am I missing?
>
> Thanks.
>
> Jerry
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Maybe you can try jvisualvm instead of JConsole.
The binary is in the /bin folder of the JAVA_HOME.
You can also add plugins to it. Maybe there is a plugin for your need.

regards,

Monitoring Tomcat JDBC Connection Pool with JConsole?

[Jerry Malcolm]

I followed the instructions to enable JMX on Tomcat.  I added the 
following lines to java config:


-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8083
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=localhost

I opened JConsole localhost;8083, and it connected.   I can see CPU, 
heap, classes, and thread graphs.  When I look at the MBean tab, I see a 
Catalina entry and a "DataSource" entry below that.  Expanding that 
gives me a ton of attributes, operations, and notifications for my 
various datasources.  That's fine.  But what I want is to be able to do 
is monitor a graph of the connection pools.  I saw an example graph on a 
3rd party web post from several years ago.  But I can't find anything 
that shows connection pool usage graphs in my JConsole.


What am I missing?

Thanks.

Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Jasper Exception when using Tomcat 9 but works with Tomcat 5.5

[Sean Brett]

Thanks for the input Chris, I’ve responded inline...

On 11/01/2018 15:48, "Christopher Schultz" 
wrote:


>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>Sean,
>
>(Thanks for moving to the mailing list; it's a much better forum for
>getting into a protracted discussion. See below for more, inline.)
>
>On 1/10/18 9:08 AM, Sean Brett wrote:
>> I've been tasked with migrating a site from one institution to
>> another. As part of the process we are attempting to update the
>> versions the site uses.
>> 
>> Initially I was asked to use Java 8 with Tomcat 5.5 (on Linux),
>> which led to issues - not to mention Tomcat 5 being 'out of
>> support¹.  I have since tried to deploy the webapps to Tomcat 9
>> with Java 8. Some of the webapps appear to run fine (and resolve
>> the issues I was having with Tomcat5). However, the main app (and 2
>> similar/smaller web apps) throws a JasperException when I go to the
>> index.jsp - the same web apps deployed to Tomcat 5.5 load without
>> issue.
>
>Moving to a later Tomcat version is best. We'll help you figure out
>these issues so you can use a current version and be better-equipped
>to stay current into the future.
>
>> Most of the similar issues I have found have suggested this would
>> be related to the jar files and possibly some sort of conflict.
>> However, the lib structure for tomcat 9 is different to tomcat5 and
>> I'm not sure which jars could be causing this. e.g.
>> https://stackoverflow.com/questions/22552244/tomcat-7-fails-to-compile
>- -jsp-
>>
>> 
>pages
>
>JAR files might conflict, but that depends upon what you have done so
>far. Easy questions:
>
>1. Have you added anything to Tomcat's lib/ directory (that would be
>CATALINA_HOME/lib or CATALINA_BASE/lib -- or both) after installing
>Tomcat 8/9?

Yes, I’ve added some jars.  Some I can’t remember why I’ve added them but
doubt they’d be connected to the problem.  Here’s a list of the
CATALINA_HOME/lib directory:
(the last 4 being the ones that have been added)

-rw-r-. 1 tomcat tomcat   36940 Sep 27 18:32 websocket-api.jar
-rw-r-. 1 tomcat tomcat  216799 Sep 27 18:32 tomcat-websocket.jar
-rw-r-. 1 tomcat tomcat  203098 Sep 27 18:32 tomcat-util-scan.jar
-rw-r-. 1 tomcat tomcat  132773 Sep 27 18:32 tomcat-util.jar
-rw-r-. 1 tomcat tomcat   34540 Sep 27 18:32 tomcat-jni.jar
-rw-r-. 1 tomcat tomcat  144927 Sep 27 18:32 tomcat-jdbc.jar
-rw-r-. 1 tomcat tomcat   42274 Sep 27 18:32 tomcat-i18n-ja.jar
-rw-r-. 1 tomcat tomcat   39391 Sep 27 18:32 tomcat-i18n-fr.jar
-rw-r-. 1 tomcat tomcat   65452 Sep 27 18:32 tomcat-i18n-es.jar
-rw-r-. 1 tomcat tomcat  251863 Sep 27 18:32 tomcat-dbcp.jar
-rw-r-. 1 tomcat tomcat  792447 Sep 27 18:32 tomcat-coyote.jar
-rw-r-. 1 tomcat tomcat   10643 Sep 27 18:32 tomcat-api.jar
-rw-r-. 1 tomcat tomcat  277812 Sep 27 18:32 servlet-api.jar
-rw-r-. 1 tomcat tomcat   61763 Sep 27 18:32 jsp-api.jar
-rw-r-. 1 tomcat tomcat   26859 Sep 27 18:32 jaspic-api.jar
-rw-r-. 1 tomcat tomcat  547928 Sep 27 18:32 jasper.jar
-rw-r-. 1 tomcat tomcat  163671 Sep 27 18:32 jasper-el.jar
-rw-r-. 1 tomcat tomcat   81575 Sep 27 18:32 el-api.jar
-rw-r-. 1 tomcat tomcat 2450404 Sep 27 18:32 ecj-4.6.3.jar
-rw-r-. 1 tomcat tomcat  285214 Sep 27 18:32 catalina-tribes.jar
-rw-r-. 1 tomcat tomcat   75123 Sep 27 18:32 catalina-storeconfig.jar
-rw-r-. 1 tomcat tomcat 1607874 Sep 27 18:32 catalina.jar
-rw-r-. 1 tomcat tomcat  119105 Sep 27 18:32 catalina-ha.jar
-rw-r-. 1 tomcat tomcat   54233 Sep 27 18:32 catalina-ant.jar
-rw-r-. 1 tomcat tomcat   18253 Sep 27 18:32 annotations-api.jar
-rw-r--r--. 1 tomcat tomcat  313898 Dec 20 13:57 dom4j-1.6.1.jar
-rw-r--r--. 1 tomcat tomcat  802494 Dec 20 13:57 freemarker-2.3.8.jar
-rw-r--r--. 1 tomcat tomcat  355030 Dec 20 13:57 mail.jar
-rw-r--r--. 1 tomcat tomcat  713037 Dec 20 14:15 postgresql-42.1.4.jar



>
>2. What JAR files are bundled with your application in WEB-INF/lib ?

-rw-r-. 1 root root   97703 Jan 11 15:05 servlet-api.jar
-rw-r-. 1 root root 8356134 Jan 11 15:05 rt.jar
-rw-r-. 1 root root  474746 Jan 11 15:05 postgresql-8.3-603.jdbc4.jar
-rw-r-. 1 root root  355030 Jan 11 15:05 mail.jar
-rw-r-. 1 root root  352668 Jan 11 15:05 log4j-1.2.8.jar
-rw-r-. 1 root root   55008 Jan 11 15:05 local-ldap-0.0.1.jar
-rw-r-. 1 root root   38114 Jan 11 15:05 ldap-src-0.0.1.jar
-rw-r-. 1 root root   38668 Jan 11 15:05 ldap.jar
-rw-r-. 1 root root   55827 Jan 11 15:05 ldap-0.0.1.jar
-rw-r-. 1 root root  146718 Jan 11 15:05 jdom.jar
-rw-r-. 1 root root  173211 Jan 11 15:05 IonoMap.jar
-rw-r-. 1 root root  457002 Jan 11 15:05 ibmjndi.jar
-rw-r-. 1 root root   38015 Jan 11 15:05 commons-logging-1.0.4.jar
-rw-r-. 1 root root   87776 Jan 11 15:05 commons-io-1.3.2.jar
-rw-r-. 1 root root   22379 Jan 11 15:05 commons-fileupload.jar
-rw-r-. 1 root root   57779 Jan 11 15:05 

Re: GC allocation failure

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Suvendu,

On 1/11/18 10:06 AM, Suvendu Sekhar Mondal wrote:
> On Tue, Jan 9, 2018 at 10:53 AM, Leon Rosenberg 
>  wrote:
>> On Mon, Jan 8, 2018 at 10:26 AM, Mark Thomas 
>> wrote:
>> 
>>> On 08/01/18 15:16, Christopher Schultz wrote:
>>> 
>>> 
>>> 
> Therefore, the first time that the GC runs, the process can
> take longer. Also, the heap is more likely to be fragmented
> and require a heap compaction. To avoid that, till now my
> strategy is to: - Start application with the minimum heap
> size that application requires - When the GC starts up, it
> runs frequently and efficiently because the heap is small
 
 I think this is a reasonable expectation for someone who
 doesn't understand the Black Art of Garbage Collection, but
 I'm not sure it's actually true. I'm not claiming that I know
 any better than you do, but I suspect that the collector
 takes its parameters very seriously, and when you introduce
 artificial constraints (such as a smaller minimum heap size),
 the GC will attempt to respect those constraints. The reality
 is that those constraints are completely unnecessary; you 
 have only imposed them because you think you know better than
 the GC algorithm.
> 
> Thank you for all your response.
> 
> Well, most of our clients are running on IBM J9 JVM and that is
> what IBM recommends :): 
> https://www.ibm.com/support/knowledgecenter/SSYKE2_9.0.0/com.ibm.java.
multiplatform.90.doc/diag/understanding/mm_heapsizing_initial.html
>
>  We have started moving our clients from WAS to Tomcat+ HotSpot
> JDK8 platform - that's why I am here, learning about it and
> throwing questions :).
> 
> One thing about memory allocation by OS: if I setup different
> values for initial and max, then after starting up the JVM,
> Windows *reserves* the max amount of memory exclusively for the
> JVM. I get that using Private Bytes counter. So that's why I
> believe there is no chance of OOM at OS level. What I am more
> interested is about the cost of heap expansion in HotSpot JVM.

The choice to pre-allocate / reserve the maximum possible heap space
is up to both the JVM and the OS. On Linux, for example, requesting
100GiB on a 64-bit platform will work right away, because Linux
doesn't actually allocate memory to the process until the process
actually tries to use it. That's why the "Linux OOM killer" is such a
big deal: the process can try to avoid it by "pre-allocating" memory,
but if it doesn't do it right, the process can still fail later when
it tries to actually use that memory.

It seems that Windows may be a little more respectful of the process's
memory requests.

In either case, pre-allocation of memory (from the OS) is not the same
thing as sizing the heap to fully-populate that memory. So, Oracle JVM
may pre-allocate the "max heap size" from the OS on startup, but then
will only size the heap to the "min heap size". Any time the JVM
decides that the heap has to grow, it will have to do a lot of work to
perform that resizing operation.

>>> Generally, the more memory available, the more efficient GC is.
>>> The general rule is you can optimise for any two of the
>>> following at the expense of the third: - low pause time - high
>>> throughout - low memory usage
>>> 
>>> It has been a few years since I listened to the experts talk
>>> about it but a good rule of thumb used to be that you should
>>> size your heap 3-5 times bigger than the minimum heap used once
>>> the application memory usages reaches steady state (i.e. the
>>> minimum value of the sawtooth on the heap usage graph)
>>> 
>>> 
>> Actually G1, which is very usable with java8 and default in jdk9,
>> doesn't produce the sawtooth graph anymore. I also think the
>> amount of memory has less influence on GC Performance in G1 or
>> Shenandoah, but instead influence if they would perform a STW
>> phase (which of course is also performance related, but
>> differently). But I am not an expert either, so I might be wrong
>> here.
>> 
>> As for OP's original statement: "When the GC starts up, it runs
>> frequently and efficiently because the heap is small", I don't
>> think it is correct anymore, especially not for G1, as long as
>> the object size is reasonable (not Humongous).
>> 
>> 
>> Leon
> 
> Yes Leon, we are seeing that G1 is works best for our app. We have 
> some large objects and we can't reduce the size immediately. So we 
> have decided to increase G1 region size for the time being and 
> collecting dead Humongous objects during Young collections.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpXi9gdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjTnw/6A0b2B4dk1sGNgUhT
8IaZ/xRaLQAHGuYzZ7WjQfNBEfaTYqwx9+to0sSCz1UEpdtHFvmiEZ9G0QkytXTS

Re: tomcat 7.0 resurrecting directory while service is installed - running in command line mode?

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Olaf,

On 1/10/18 12:50 PM, Olaf Kock wrote:
> 
> 
> On 10.01.2018 17:57, Christoph P.U. Kukulies wrote:
>> Am 10.01.2018 um 17:06 schrieb Christoph P.U. Kukulies:
>>> The problem is overlaid by the following:
>>> 
>>> [2018-01-10 16:35:37] [info]  [ 6340] Commons Daemon procrun 
>>> (1.0.15.0 64-bit) s tarted [2018-01-10 16:35:37] [info]  [
>>> 6340] Running 'Tomcat7' Service... [2018-01-10 16:35:37] [info]
>>> [ 6376] Starting service... [2018-01-10 16:35:37] [error] [
>>> 6376] Failed creating java C:\Program Files\Java 
>>> \jdk1.8.0_111\jre\bin\server\jvm.dll [2018-01-10 16:35:37]
>>> [error] [ 6376] Das System kann den angegebenen Pfad nicht 
>>> finden. [2018-01-10 16:35:37] [error] [ 6376] ServiceStart
>>> returned 1 [2018-01-10 16:35:37] [error] [ 6376] Das System
>>> kann den angegebenen Pfad nicht finden. [2018-01-10 16:38:06]
>>> [info]  [ 6340] Run service finished. [2018-01-10 16:38:06]
>>> [info]  [ 6340] Commons Daemon procrun finished
>>> 
>>> 
>>> My system wide setting of JAVA_HOME is (installed recent jre
>>> today since my Java instalation jdk1.8.0.111 was hosed and the
>>> error message is correct in this sense).
>>> 
> AFAIK the batch files are completely insignificant when services
> are started on Windows.

Correct, except that the batch files can be used to install the
service, e.g.:

C:\path> SET CATALINA_HOME=...
C:\path> SET CATALINA_BASE=...
C:\path> %CATALINA_HOME%\bin\service.bat install

(from memory... it's something like that)

The point is that the batch scripts use environment variable values
that were set at the time of execution to set all of the command-line
parameters that the service will use when it's actually launched.
Changing the environment variable values after service.bat is run does
not have any effect on the service (unless you "reinstall" it).

> Linux Daemons often use the shellscripts, but Windows - as you
> probably figured - uses the configuration it finds in its registry,
> to be configured through tools like tomcat7w.exe

Correct.

> And indeed (as you asked earlier): I meant enclosing everything in 
> double quotes as in SET "key=value with spaces" - but that's from
> memory as I don't have any Windows system available for reference.

At this point, most of the batch scripts behave "properly" when paths
contain spaces. But things are slightly worse on Windows when it comes
to spaces in paths for at least two reasons:

1. Some commands or utilities understand spaces and don't necessarily
need quoting around the paths, which is enormously confusing to a
UNIX-type person like me.

2. Windows capabilities to quote/unquote strings are often
unpredictable -- again especially for UNIX-y folks like me -- do
sometimes it's tough to make sure that 100% of possible cases are covere
d.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpXiTodHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFhYLxAAs89p2tYykgCiuiCm
xuowKgzZrTKlCUl3vxBCzLQFZVUif2uY2Qj4Trj5/pb4ukDx9YmZKQ9HwExcR0Ww
E8DpZ/LR704SQIwyxlX3YCFjeIcficwzKxlDLEE4P2iMOWxTXTkWGNFnb3MjrN57
3TbO11UGLbuUOiA216PzyyzN880YPPI7x+UkgGS6/OLPdvugQlNZ6HeAFfe5H3Ix
aWgwM61iWA0/IGjws9qYpBpyaISUa6/s5MuMDyvhMbYfyn8V3MPPKaQgLtCfeWz2
iK9A7AN++55/T/0O7CXn4GKFE+oWpmWyfeMrYPmX+yVyTqCCOvr/HPkU70fEh8kZ
CWx1Ml9MQ2m9GwAa5XxFyik9/eJGa3ATUS80tmANup5G816HztGhFDmOLfxe1HW8
yjyA8Wz0v56xIDHVUj6itqj9e9en9RjqRbuwjOceOS+m/CwUWFa/qtYMbJH0M8N1
l/OoaSrZV08OwZIT/A/PseAesMXRpd4zOhvu+9W0zEwNl0krxtKW9JRkl3C41F0L
QZ6hrph+mJL5jSYdW4ZOFpXuwojgBKlos4t9PG3+h1bLJEunugGI0zuOGSqMHald
OwLwI4wK+4XlXDe9iw1nsXk8N2NwkzI1SvFy3EPeSvczAXY1briOIUZ3CIT2csDn
/4Wk3yP43vVJkl+UqspArdCUvAw=
=WeK4
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Jasper Exception when using Tomcat 9 but works with Tomcat 5.5

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Sean,

(Thanks for moving to the mailing list; it's a much better forum for
getting into a protracted discussion. See below for more, inline.)

On 1/10/18 9:08 AM, Sean Brett wrote:
> I've been tasked with migrating a site from one institution to
> another. As part of the process we are attempting to update the
> versions the site uses.
> 
> Initially I was asked to use Java 8 with Tomcat 5.5 (on Linux),
> which led to issues - not to mention Tomcat 5 being 'out of
> support¹.  I have since tried to deploy the webapps to Tomcat 9
> with Java 8. Some of the webapps appear to run fine (and resolve
> the issues I was having with Tomcat5). However, the main app (and 2
> similar/smaller web apps) throws a JasperException when I go to the
> index.jsp - the same web apps deployed to Tomcat 5.5 load without
> issue.

Moving to a later Tomcat version is best. We'll help you figure out
these issues so you can use a current version and be better-equipped
to stay current into the future.

> Most of the similar issues I have found have suggested this would
> be related to the jar files and possibly some sort of conflict.
> However, the lib structure for tomcat 9 is different to tomcat5 and
> I'm not sure which jars could be causing this. e.g. 
> https://stackoverflow.com/questions/22552244/tomcat-7-fails-to-compile
- -jsp-
>
> 
pages

JAR files might conflict, but that depends upon what you have done so
far. Easy questions:

1. Have you added anything to Tomcat's lib/ directory (that would be
CATALINA_HOME/lib or CATALINA_BASE/lib -- or both) after installing
Tomcat 8/9?

2. What JAR files are bundled with your application in WEB-INF/lib ?

> I've also tried changing the index.jsp into a simpler (HelloWorld)
> form - removing any code. And tried deleting the 'work' directory
> before restarting Tomcat 
> (http://grokbase.com/t/tomcat/users/072v2kf60h/java-permission-denied-
error
>
> 
- -in-tomcat).  In both cases the exception is still thrown.

Can you post the full text of HelloWorld.jsp?

I'm wondering if you have some odd syntax or are expecting to use a
"custom base JSP servlet class" or something like that which is no
longer compatible with Tomcat's current JSP servlet. If so, we'll try
to figure out why the custom code was necessary and make arrangements
to replace it.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpXh1QdHGNocmlzQGNo
cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFiFnA//ebj5XoXFXG0vEAIA
DVSR2X8JwfeiW5Fiamt1h1jw1a/zl92Q9TBsqJ4ocxceipules1+us7Tpi0QyURt
OozqydqC8+qm7N/m/LBPk8Mxo/dMgV7s5Yk2bITlQFbPzKrlkgfL2YKDcbsNlBWs
RZxUdVD11GkdDp9DGUDdoYcMjYW+nDhHr9NLjlKvpyCUZ1lwJ68uSPahzJL6/ba+
FNiXsghDNGi7aZOh8P0Hy3A/5pYhdugcU8M1HYtMbdGrmqaKGB9afjGT7Odp5Eo+
38bIc5wWJZN9Ak6IVzi3Avloa3STckDuzGlWkTS82q9cpG8uy461F2MHC0yjZha6
+rZz299/QZ2Ouzbc2ICUxJLWsQvSfWii9zdAS8y6+LevmyFzpgbTdTsQfOPicGtc
+Ow9VWl2SJe+sPNFkgOdKP+XkKTOPUJjdP+hhjOzUPpEjJ71Hcf5x8bZ1EuceAt4
ORnM9LCFY4ecSIAhmMivdajxHHHf2IWvCnI4emaTjlULUiyq70go9bBAjiTd2k18
oMsr1gIXgWonp3Gl3RW7/o/fXilaPIOJeEeKPEKgPtvEFf56EkkAouPo/KXiDIMF
CJMDPGSPx2n7ITZcFJnT0vIZFIJh9fZqUuxAnqG/f5A9cN7QGQf2sORKUgWMOZtp
5G743rnFC/IQE3t1irkHxn/yK0s=
=W9Le
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: GC allocation failure

[Suvendu Sekhar Mondal]

On Tue, Jan 9, 2018 at 10:53 AM, Leon Rosenberg
 wrote:
> On Mon, Jan 8, 2018 at 10:26 AM, Mark Thomas  wrote:
>
>> On 08/01/18 15:16, Christopher Schultz wrote:
>>
>> 
>>
>> >> Therefore, the first time that the GC runs, the process can take
>> >> longer. Also, the heap is more likely to be fragmented and require
>> >> a heap compaction. To avoid that, till now my strategy is to: -
>> >> Start application with the minimum heap size that application
>> >> requires - When the GC starts up, it runs frequently and
>> >> efficiently because the heap is small
>> >
>> > I think this is a reasonable expectation for someone who doesn't
>> > understand the Black Art of Garbage Collection, but I'm not sure it's
>> > actually true. I'm not claiming that I know any better than you do,
>> > but I suspect that the collector takes its parameters very seriously,
>> > and when you introduce artificial constraints (such as a smaller
>> > minimum heap size), the GC will attempt to respect those constraints.
>> > The reality is that those constraints are completely unnecessary; you
>> > have only imposed them because you think you know better than the GC
>> > algorithm.

Thank you for all your response.

Well, most of our clients are running on IBM J9 JVM and that is what
IBM recommends :):
https://www.ibm.com/support/knowledgecenter/SSYKE2_9.0.0/com.ibm.java.multiplatform.90.doc/diag/understanding/mm_heapsizing_initial.html

We have started moving our clients from WAS to Tomcat+ HotSpot JDK8
platform - that's why I am here, learning about it and throwing
questions :).

One thing about memory allocation by OS: if I setup different values
for initial and max, then after starting up the JVM, Windows
*reserves* the max amount of memory exclusively for the JVM. I get
that using Private Bytes counter. So that's why I believe there is no
chance of OOM at OS level. What I am more interested is about the cost
of heap expansion in HotSpot JVM.

>> Generally, the more memory available, the more efficient GC is. The
>> general rule is you can optimise for any two of the following at the
>> expense of the third:
>> - low pause time
>> - high throughout
>> - low memory usage
>>
>> It has been a few years since I listened to the experts talk about it
>> but a good rule of thumb used to be that you should size your heap 3-5
>> times bigger than the minimum heap used once the application memory
>> usages reaches steady state (i.e. the minimum value of the sawtooth on
>> the heap usage graph)
>>
>>
> Actually G1, which is very usable with java8 and default in jdk9, doesn't
> produce the sawtooth graph anymore.
> I also think the amount of memory has less influence on GC Performance in
> G1 or Shenandoah, but instead influence if they would perform a STW phase
> (which of course is also performance related, but differently).
> But I am not an expert either, so I might be wrong here.
>
> As for OP's original statement: "When the GC starts up, it runs frequently
> and
> efficiently because the heap is small", I don't think it is correct
> anymore, especially not for G1, as long as the object size is reasonable
> (not Humongous).
>
>
> Leon

Yes Leon, we are seeing that G1 is works best for our app. We have
some large objects and we can't reduce the size immediately. So we
have decided to increase G1 region size for the time being and
collecting dead Humongous objects during Young collections.


Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org