Re: Tomcat9 not listening to ipv4 port 8080, only ipv6

[Suvendu Sekhar Mondal]

Hello Christoph,

On Tue, Nov 28, 2023, 5:55 PM Christoph Kukulies 
wrote:

> I'm pulling my hairs on a suddenly occured - possibly - misconfiguration.
> But I can't find it out:
>
> catalina.2023-11-28.log:
>
>
> 28-Nov-2023 13:15:43.742 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server version name:
>   Apache Tomcat/9.0.58 (Ubuntu)
> 28-Nov-2023 13:15:43.743 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server built:
>   Jan 6 1970 15:09:28 UTC
> 28-Nov-2023 13:15:43.744 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server version
> number: 9.0.58.0
> 28-Nov-2023 13:15:43.744 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log OS Name:
>   Linux
> 28-Nov-2023 13:15:43.744 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log OS Version:
>   5.15.0-89-generic
> 28-Nov-2023 13:15:43.745 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Architecture:
>   amd64
> 28-Nov-2023 13:15:43.745 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Java Home:
>   /usr/lib/jvm/java-11-openjdk-amd64
> 28-Nov-2023 13:15:43.745 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
>   11.0.20.1+1-post-Ubuntu-0ubuntu122.04
> 28-Nov-2023 13:15:43.745 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
>   Ubuntu
> 28-Nov-2023 13:15:43.746 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
>   /var/lib/tomcat9
> 28-Nov-2023 13:15:43.746 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
>   /usr/share/tomcat9
> 28-Nov-2023 13:15:43.758 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: --add-opens=java.base/java.lang=ALL-UNNAMED
> 28-Nov-2023 13:15:43.759 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: --add-opens=java.base/java.io=ALL-UNNAMED
> 28-Nov-2023 13:15:43.759 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: --add-opens=java.base/java.util=ALL-UNNAMED
> 28-Nov-2023 13:15:43.760 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
> 28-Nov-2023 13:15:43.760 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
> 28-Nov-2023 13:15:43.760 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument:
> -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
> 28-Nov-2023 13:15:43.761 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> 28-Nov-2023 13:15:43.761 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.awt.headless=true
> 28-Nov-2023 13:15:43.761 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djdk.tls.ephemeralDHKeySize=2048
> 28-Nov-2023 13:15:43.761 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> 28-Nov-2023 13:15:43.762 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> 28-Nov-2023 13:15:43.762 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dignore.endorsed.dirs=
> 28-Nov-2023 13:15:43.762 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dcatalina.base=/var/lib/tomcat9
> 28-Nov-2023 13:15:43.762 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Dcatalina.home=/usr/share/tomcat9
> 28-Nov-2023 13:15:43.763 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.io.tmpdir=/tmp
> 28-Nov-2023 13:15:43.768 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache
> Tomcat Native library [1.2.31] using APR version [1.7.0].
> 28-Nov-2023 13:15:43.769 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false], random
> [true], UDS [true].
> 28-Nov-2023 13:15:43.771 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL
> configuration: useAprConnector [false], useOpenSSL [true]
> 28-Nov-2023 13:15:43.776 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 3.0.2 15 Mar 2022]
> 28-Nov-2023 13:15:44.229 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio-8080"]
> 

Re: Tomcat 10.1.15 JVM crashes randomly on startup

[Suvendu Sekhar Mondal]

Hello Øyvind,

> While I'm waiting for my hosting provider to disable the Sentinel Agent, I'm 
> attaching a full crash report below, from another crash this morning.

That's a good idea. After removal if JVM is not crashing abruptly,
you'll be certain that Sentinel agent is the problem.

You can also try to extract thread details out of the crash dump and
look at the "https-openssl-nio-0.0.0.0-443-exec-27" thread. jstack can
help. Command will be something like: jstack -J-d64 JAVA_HOME\bin\java
E:\tomcat10\hs_err_pid15260.mdmp
Please use the same JRE which produced the dump to avoid any
incompatibility issue.

On Tue, Nov 14, 2023 at 3:28 PM Øyvind Flatval
 wrote:
>
> > From: Mark Thomas 
> > Sent: Monday, November 13, 2023 09:15
> > To: users@tomcat.apache.org 
> > Subject: Re: Tomcat 10.1.15 JVM crashes randomly on startup
> >
> > On 13/11/2023 07:52, Øyvind Flatval wrote:
> > > Greetings!
> > >
> > > We are currently experiencing a very vague problem with our Tomcat 10.1 
> > > instance, where the JVM will crash almost instantly after Tomcat is done 
> > > starting up.
> > > The problem happens somewhat regularly, and only happens within the first 
> > > minute after starting Tomcat. The solution is always to just attempt to 
> > > restart Tomcat and this usually only take 1 attempt.
> > >
> > >
> > > Tomcat is running a single web application, that sees traffic over both 
> > > Websocket (several 100 clients) and Https (maybe 20-40 users during peak 
> > > + a few API clients)
> > >
> > > Our setup is a somewhat old VM.
> > > Windows Server 2016 Standard 1607, buil 14343.6351
> > > Intel Xeon E5-2680 v4 @ 2.40 GHz
> > > 4 x Cores
> > > 8 GB Memory
> > >
> > > JRE version: OpenJDK Runtime Environment Temurin-17.0.8.1+1 (17.0.8.1+1) 
> > > (build 17.0.8.1+1)
> > > Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (17.0.8.1+1, mixed 
> > > mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, 
> > > windows-amd64)
> > >
> > > Tomcat version is 10.1.15 64 bit
> > >
> > > The company we rent our VM servers from has installed Sentinel One that 
> > > connects to the JVM through an agent. I have very little knowledge of 
> > > what if anything this can do. Since this is the only thing out of the 
> > > ordinary running on this setup, my fear is that this might be causing the 
> > > crashes.
> >
> > Some web searching suggests that is certainly likely.
> >
> > > I include a partial example of the exception log here, some data may have 
> > > been removed for privacy reasons. Please let med know if I should include 
> > > the full file, since I've removed parts of it due to pure length.
> >
> > The full file might contain a few more hints as to what went wrong.
> >
> > Mark
> >
>
> While I'm waiting for my hosting provider to disable the Sentinel Agent, I'm 
> attaching a full crash report below, from another crash this morning.
>
> >
> > >
> > > #
> > > # A fatal error has been detected by the Java Runtime Environment:
> > > #
> > > #  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x1131b169, 
> > > pid=2084, tid=5160
> > > #
> > > # JRE version: OpenJDK Runtime Environment Temurin-17.0.8.1+1 
> > > (17.0.8.1+1) (build 17.0.8.1+1)
> > > # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (17.0.8.1+1, mixed 
> > > mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, 
> > > windows-amd64)
> > > # Problematic frame:
> > > # j  
> > > jakarta.servlet.http.HttpServletRequestWrapper.getDateHeader(Ljava/lang/String;)J+1
> > > #
> > > # Core dump will be written. Default location: 
> > > E:\tomcat10\hs_err_pid2084.mdmp
> > > #
> > > # If you would like to submit a bug report, please visit:
> > > #   https://github.com/adoptium/adoptium-support/issues
> > > #
> > >
> > > ---  S U M M A R Y 
> > >
> > > Command Line: -Dcatalina.home=E:\tomcat10 -Dcatalina.base=E:\tomcat10 
> > > -Dignore.endorsed.dirs=E:\tomcat10\endorsed 
> > > -Djava.io.tmpdir=E:\tomcat10\temp 
> > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
> > > -Djava.util.logging.config.file=E:\tomcat10\conf\logging.properties 
> > > -Djava.locale.providers=COMPAT 
> > > --add-opens=java.base/java.lang=ALL-UNNAMED 
> > > --add-opens=java.base/java.io=ALL-UNNAMED 
> > > --add-opens=java.base/java.util=ALL-UNNAMED 
> > > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED 
> > > --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED exit abort -Xms768m 
> > > -Xmx5090m -agentpath:C:\Program Files\SentinelOne\Sentinel Agent 
> > > 22.3.4.612\SentinelJava64.dll
> > >
> > > Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 4 cores, 7G,  Windows 
> > > Server 2016 , 64 bit Build 14393 (10.0.14393.5786)
> > > Time: Sat Nov 11 18:18:30 2023 W. Europe Standard Time elapsed time: 
> > > 153.828934 seconds (0d 0h 2m 33s)
> > >
> > > ---  T H R E A D  ---
> > >
> > > Current thread (0x2e6a9bd0):  JavaThread 
> > > 

Re: Are there any known class loader leaks in Tomcat 9?

[Suvendu Sekhar Mondal]

Hello William,

On Tue, Nov 7, 2023 at 4:29 PM William Crowell
 wrote:
>
> Olaf and Sevendu,
>
> Thank you for your replies.  Correct, I sincerely doubt this is a Tomcat 
> class loading bug.
>
> I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into 
> memory, and it is a single class loader.
>
> I am going to periodically run: jcmd  GC.class_stats
>
> I am only having the issue in general operation and not on a redeploy, and I 
> have to restart the server daily.  I will find out more details as to how 
> these classes are loaded into memory.
>
> Here is what is going on…
>
> I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on 
> it.  It is running JDK 1.8.0_371-b25.  I have min and max JVM heap setting at 
> 8GB.  JVM arguments are:
>
> -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC 
> -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 
> -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 
> -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 
> -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 
> -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC 
> -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime 
> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps 
> -XX:+UseCompressedClassPointers -XX:+UseCompressedOops 
> -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
>
> The MaxMetaspaceSize is not set, so this means it is unlimited.
>
> After the server comes up and after a short period of time I get a native out 
> of memory condition:
>
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 8589934592 bytes for 
> committing reserved memory.
> # Possible reasons:
> #   The system is out of physical RAM or swap space
> #   The process is running with CompressedOops enabled, and the Java Heap may 
> be blocking the growth of the native heap
> # Possible solutions:
> #   Reduce memory load on the system
> #   Increase physical memory or swap space
> #   Check if swap backing store is full
> #   Decrease Java heap size (-Xmx/-Xms)
> #   Decrease number of Java threads
> #   Decrease Java thread stack sizes (-Xss)
> #   Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> #  Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700
> #
> # JRE version:  (8.0_371) (build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode 
> linux-amd64 compressed oops)
> # Core dump written. Default location: /hosting/core or core.191803
> #
>

So, it is not a Metaspace OOM. It can happen that Metaspace or some
other segment of JVM is increasing with time. Since you have NMT
enabled(-XX:NativeMemoryTracking=detail), please collect native memory
stats periodically until JVM crashes. After that, if you compare them,
you should see one of the segments is growing with time. Then you can
focus on that area. jcmd  VM.native_memory can help with this.


> Regards,
>
> William Crowell
>
>
> From: Olaf Kock 
> Date: Tuesday, November 7, 2023 at 4:25 AM
> To: users@tomcat.apache.org 
> Subject: Re: Are there any known class loader leaks in Tomcat 9?
>
> On 06.11.23 18:55, William Crowell wrote:
> > Good afternoon,
> >
> > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am 
> > loading some very large Java classes into Metaspace.  I know this is not 
> > good practice, but I inherited this library.  These classes have business 
> > rules and are doing some basic primitive and array manipulations, and I am 
> > running out of native memory.  I don’t think there are any JNI calls in 
> > this code base.
> >
> > Are there anything existing issues with the Tomcat 9 class loader?  I doubt 
> > there are but wanted to check.
>
> Hi William,
>
> when you say "I am loading some very large Java classes into Metaspace.
> I know this is not good practice" it does not sound like this inherited
> library is using tomcat's standard way of classloading - please clarify.
>
> Also, nothing in that statement points to anything where a leak on
> tomcat's side would make a difference: Are you experiencing problems
> after reloading a webapp, or just in general operation? You might simply
> need to assign more metaspace to the VM for this library to operate
> properly.
>
> So, this boils down to:
>
> * How does this library load the classes into memory?
>
> * Do you redeploy?
>
> * What's your metaspace (and general memory) setting, and what are the
> conditions under which you run out of memory?
>
> * Do you see any log message that hint at classes that won't be garbage
> collected because of stale references? Tomcat issues those warnings, in
> case you (for example) start your own background threads that keep
> holding the memory allocated to them.
>
> Also: I second your doubt about Tomcat's 

Re: Are there any known class loader leaks in Tomcat 9?

[Suvendu Sekhar Mondal]

Hello William,

On Mon, Nov 6, 2023 at 11:25 PM William Crowell
 wrote:
>
> Good afternoon,
>
> I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am 
> loading some very large Java classes into Metaspace.  I know this is not good 
> practice, but I inherited this library.  These classes have business rules 
> and are doing some basic primitive and array manipulations, and I am running 
> out of native memory.  I don’t think there are any JNI calls in this code 
> base.
>
How are you loading those large classes? using some custom classloader
OR Tomcat's normal classloader? Also, are you using a single
classloader to load all/multiple classes? Reason I asked is, until ALL
classes loaded by a classloader are de-referenced, the entire set of
classes loaded by the classloader will NOT be garbage collected. Most
possibly that is what is happening in your case.

Also, how fast Metaspace is growing? I will suggest checking the
contents of Metaspace by taking class stats details periodically.
GC.class_stats of Jcmd is helpful.

> Are there anything existing issues with the Tomcat 9 class loader?  I doubt 
> there are but wanted to check.
>

I have been using Tomcat 9 for the last two years and have not
experienced any problem where Tomcat is holding onto classes.

> Regards,
>
> William Crowell
>
>
> This e-mail may contain information that is privileged or confidential. If 
> you are not the intended recipient, please delete the e-mail and any 
> attachments and notify us immediately.
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re:

[Suvendu Sekhar Mondal]

On Mon, Nov 6, 2023 at 3:15 PM Mark Thomas  wrote:
>
> On 05/11/2023 17:23, Greg Huber wrote:
> > Thanks Mark and Chris.
> >
> >>>  >>>   base="/home/devuser/git/myproject/target/classes"
> >>> className="org.apache.catalina.webresources.DirResourceSet"
> >>>   webAppMount="/WEB-INF/classes" />
> >
> > I have not noticed any slowness yet.
> >
> > There are alot of jars (approx 160), but the target/classes folder are my
> > app's classes that I am working on.  These can change (ie not static), so
> > may be better to switch it off.
> >
> > Is there anyway to calculate the size needed for the cache setting?
>
> The maximum useful size will be the total size of static resources (i.e.
> everything NOT under WEB-INF/lib or WEB-INF/classes).
>
> The right size is going to be a trade-off between the cost of the memory
> for the cache and the benefits the cache brings. Those benefits are
> going to be application (and hardware) dependent.
>
> Mark

+1

You can also explore the combination of cacheTtl and cache size. By
default, TTL is 5 sec. If the cache is not full, a longer TTL means
better performance BUT if cache is getting filled up because of longer
TTL then tomcat will evict elements even before TTL elapsed. Again,
it's a bit tricky.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.76 Memory leak with Java 17

[Suvendu Sekhar Mondal]

Hi Chris,

On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
 wrote:
>
> James,
>
> On 7/11/23 10:21, James Boggs wrote:
> > We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
> > Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
> >
> > We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
> > ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
> > csr file, and imported a new SSL certificate from the CA into the new
> > keystore.
> >
> > The website works but after logging in there are memory leak warnings
> > and the Tomcat service crashes within just a couple of minutes.
> >
> > We even upgraded to 9.0.76 and the issue persists.
> >
> > Below is an excerpt from the stderr log file.
> >
> > I have been unable to find any recent threads on this, any help is
> > appreciated. Is any other information needed?
>
> I think you have included all necessary information. I'm chopping-out
> the irrrelevant bits:
>
> > 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
> > registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
> > unregister it when the web application was stopped. To prevent a memory
> > leak, the JDBC Driver has been forcibly unregistered.
>
> This is actually "okay" in that Tomcat has detected a global JDBC driver
> registration performed by the application, and is fixing the problem for
> you. The application, however, is making a mistake by not de-registering
> that JDBC driver. Your options are to move the driver library from your
> application into Tomcat's lib/ directory, fix the application to
> de-register the driver when it shuts down, or just ignore the warning.
> But you should fix the application.
>
> > 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
> > appears to have started a thread named
> > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] 
> > but has failed to stop it. This is very likely to create a memory leak. 
> > Stack trace of thread:
>
> There are multiple instances of this same message and THIS is your
> problem. The problem is what the error message says: your application
> has started a Thread and never stopped it. The "memory leak" comes from
> the fact that the Thread has inherited the web application's ClassLoader
> and the web application is being re-loaded. When that happens, Tomcat
> discards the ClassLoader which usually means the GC will clean up after
> it at some point in the future. But that Thread is still running and
> will keep the ClassLoader in memory, likely forever.
>
> You have a few options:
>
> 1. Fix the application. The application needs to shut-down any threads
> is starts during its operation, preferrably in a
> ServletContextListener's destroy method or similar.
>
> 2. Don't hot-reload your application. Instead, shut-down the JVM and
> re-start it. Ovbviously, this may have availability implications, but
> then again so does running out of memory and having to bounce the JVM,
> anyway.
>

I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.

> 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler 
> ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds
> 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler 
> ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T21:35:40.495Z INFOStopping service [Catalina]
> 2023-07-10T21:35:40.910Z INFODestroyed pool : 
> |default|lo|-2023-07-10T20-51-53.099285500Z

Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.

> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dconfig.url=D:\ords222
> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xms1024M
> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xmx1024M

> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> exit
> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> abort
> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xms128m
> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xmx256m



> > 2023-07-10T21:35:40.944Z SEVERE  The web application [rplans-vpd]
> > created a ThreadLocal with key of type
> > [java.lang.ThreadLocal.SuppliedThreadLocal] (value
> > 

Re: Core Dump File generating Issue

[Suvendu Sekhar Mondal]

Hello Mohit,

On Tue, Jun 6, 2023 at 4:48 AM Chaudhary, Mohit  wrote:
>
> Hello Team,
>
> Onto our tomcat production environment we are having two server clustered 
> .where we have only one tomcat app running.
> We are facing some issue in production system and observed there are core 
> dump files getting generated inside the tomcat-install/bin folder. The core 
> is getting generated along with pid eg. core.21539.
> Some times,core dump file keeps up growing in larger size amount like 18GB to 
> 19GB which is making full the allocated disk for tomcat-install/bin file 
> system. This intern causes to get tomcat application on hung state and result 
> an instance down for production server. Some time it is highly impacting to 
> one node of cluster and some time both the node were down.
> For tomcat side, we have not configured/enabled anything related to generate 
> the core dump  file at tomcat-install/bin location. We are not aware from 
> where and how this core file is getting generated.

Tomcat does not generate core dumps. Your JVM is producing it as it's
crashing due to some issue. Normally when JVM crashes due to fatal
error, apart from generating core dumps it also generate one crash log
file(e.g. hs_err_pid.log). It has some human readable information
about why the crash occurred. Are you getting it?

> The client is asking for an evidence from where the core file is getting 
> generated and also application details etc.
> We require your support how shall to proceed to analyze core file and provide 
> evidence to client. Currently ,we do not have any analyzer tool install on 
> system.

You have not provided any environment information. So, it'll be hard
to tell what tools you can use.

There are a couple of ways to extract information out of a core dump.
To extract thread details I generally use jstack. Command will be
something like this:
 jstack -J-d64 path_to_java/bin/java path_of_your_core_dump/core.21539

You can also use jmap to create a heap dump out of a java core. Sample
command: jmap -J-d64 path_to_java/bin/java
path_of_your_core_dump/core.21539

Please note that path_to_java should be the Java binary location which
produced the core dump, otherwise it throws errors and data extraction
fails. That means, you have to use those JDK tools in your production
system. Also note, since you'll be processing a 18GB core dump, those
tools can consume a good amount of CPU and memory.

>
> Thanks & Regards,
> Mohit Chaudhary
>
>
>
> DXC Technology Company -- This message is transmitted to you by or on behalf 
> of DXC Technology Company or one of its affiliates. It is intended 
> exclusively for the addressee. The substance of this message, along with any 
> attachments, may contain proprietary, confidential or privileged information 
> or information that is otherwise legally exempt from disclosure. Any 
> unauthorized review, use, disclosure or distribution is prohibited. If you 
> are not the intended recipient of this message, you are not authorized to 
> read, print, retain, copy or disseminate any part of this message. If you 
> have received this message in error, please destroy and delete all copies and 
> notify the sender by return e-mail. Regardless of content, this e-mail shall 
> not operate to bind DXC Technology Company or any of its affiliates to any 
> order or other contract unless pursuant to explicit written agreement or 
> government initiative expressly permitting the use of e-mail for such purpose.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Need to know about analyzing of thread dump and heap dump

[Suvendu Sekhar Mondal]

On Tue, Apr 4, 2023, 5:08 PM  wrote:

>
>
>
> > -Original Message-
> > From: Suvendu Sekhar Mondal 
> > Sent: Tuesday, April 04, 2023 1:54 AM
> > To: Tomcat Users List 
> > Subject: Re: Need to know about analyzing of thread dump and heap dump
> >
> > Hi Koustav,
> >
> > On Tue, Apr 4, 2023, 1:28 AM Naha, Koustav 
> > wrote:
> >
> > > Hi all,
> > >
> > > Good day.
> > >
> > > Can someone suggest me some good tools to analyze heap dump and
> > thread
> > > dumps which we can use in real time production environment.
> > > Also, GUI based tools will be a good one to use.
> > >
> >
> > Pease don't use any online third party tool to analyze heap dumps.
> Because a
> > memory snapshot contains all the details including your secrets :).
> IMHO, it's
> > better to use offline tools like Eclipse MAT, IBM's heap dump analyzer
> for
> > heap dump analysis.
> >
> > For thread dumps analysis, in most cases I found that multiple thread
> dumps
> > are required to identify slowness or establish a pattern. I haven't came
> across
> > any thread dumps analyzer which takes multiple thread dumps as input.
> > That's why I always go back to IBM thread dumps analyzer.
> >
> > Both of these are free to download.
> >
> >
> >
> > > Please pour in your 2 cents.
> > >
> > > Thanks and Regards,
> > > Koustav Naha
> > >
> > >
> > > DXC Technology Company -- This message is transmitted to you by or on
> > > behalf of DXC Technology Company or one of its affiliates. It is
> > > intended exclusively for the addressee. The substance of this message,
> > > along with any attachments, may contain proprietary, confidential or
> > > privileged information or information that is otherwise legally exempt
> > > from disclosure. Any unauthorized review, use, disclosure or
> > > distribution is prohibited. If you are not the intended recipient of
> > > this message, you are not authorized to read, print, retain, copy or
> > > disseminate any part of this message. If you have received this
> > > message in error, please destroy and delete all copies and notify the
> > > sender by return e-mail. Regardless of content, this e-mail shall not
> > > operate to bind DXC Technology Company or any of its affiliates to any
> > > order or other contract unless pursuant to explicit written agreement
> > > or government initiative expressly permitting the use of e-mail for
> such
> > purpose.
> > >
>
>
> FastThread analyzes more than one dump and compares them.
>

Thanks, John! Need to try it out.

>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Need to know about analyzing of thread dump and heap dump

[Suvendu Sekhar Mondal]

Hi Koustav,

On Tue, Apr 4, 2023, 1:28 AM Naha, Koustav  wrote:

> Hi all,
>
> Good day.
>
> Can someone suggest me some good tools to analyze heap dump and thread
> dumps which we can use in real time production environment.
> Also, GUI based tools will be a good one to use.
>

Pease don't use any online third party tool to analyze heap dumps. Because
a memory snapshot contains all the details including your secrets :). IMHO,
it's better to use offline tools like Eclipse MAT, IBM's heap dump analyzer
for heap dump analysis.

For thread dumps analysis, in most cases I found that multiple thread dumps
are required to identify slowness or establish a pattern. I haven't came
across any thread dumps analyzer which takes multiple thread dumps as
input. That's why I always go back to IBM thread dumps analyzer.

Both of these are free to download.



> Please pour in your 2 cents.
>
> Thanks and Regards,
> Koustav Naha
>
>
> DXC Technology Company -- This message is transmitted to you by or on
> behalf of DXC Technology Company or one of its affiliates. It is intended
> exclusively for the addressee. The substance of this message, along with
> any attachments, may contain proprietary, confidential or privileged
> information or information that is otherwise legally exempt from
> disclosure. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient of this message, you are
> not authorized to read, print, retain, copy or disseminate any part of this
> message. If you have received this message in error, please destroy and
> delete all copies and notify the sender by return e-mail. Regardless of
> content, this e-mail shall not operate to bind DXC Technology Company or
> any of its affiliates to any order or other contract unless pursuant to
> explicit written agreement or government initiative expressly permitting
> the use of e-mail for such purpose.
>

Re: tomcat is not coming Up

[Suvendu Sekhar Mondal]

Hello Prabu,

On Wed, Jan 11, 2023, 8:56 PM Ganesan, Prabu
 wrote:

> Hi team
>
>
>
> Our Production Servers are down, Not Coming up When we are trying to start
> the services
>
Which OS are you using? Please provide some basic details ( like OS, Tomcat
version, JRE version ) about your environment so that this community can
provide some help.

>
>
> In logs we couldn’t see Any Errors only I can see Info.
>
>
>
These logs are normal but indicates change in APR version.

>
>
> Though I can see some changes in this. But we have not made any changes
>
>
>
>
>
> *Old :* INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Djava.library.path=/usr/local/apr/lib
>
>
>
> *New : *INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Xmx3072m
>
>
>
>
> *Old : *INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.23] using APR version [1.6.3]
>
>
>
> *New :* INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.17] using APR version [1.4.8]
>
>
>
>
>
>
>
>
>
> Could you please confirm. Is this could be the reason to not start up?
>
>
>
> We have not done any changes from our end.
>
>
>
>
>
> Please help us what more required I ll Provide logs
>
>
>
> Thanks & Regards,
>
> _[image:
> Email_CBE.gif]
>
> *PrabuGanesan*
>
> *Consultant|MS-Nordics*
>
> capgemini India Pvt. Ltd. | Bangalore
>
> Contact: +91 8526554535
>
> Email: prabhu.c.gane...@capgemini.com
>
>
>
> www.capgemini.com
>
> *People matter, results count.*
>
> __
>
> *Connect with Capgemini:*
> 
>  
> 
> 
> 
>
>
>
> Please consider the environment and do not print this email unless
> absolutely necessary.
>
> Capgemini encourages environmental awareness.
>
>
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>

Re: About granting permissions to Tomcat JVM

[Suvendu Sekhar Mondal]

Hello Martin,

On Sun, Oct 9, 2022 at 7:03 PM Martin Moore  wrote:
>
> the ProcessExplorer shows that a Java process is running on the file and
> this only after actually performing the delete from Java.
>
Sometimes Logstash gives me trouble like this.

Most probably some part of the app was still holding reference to the
file. Can you please ensure your app is closing all input/output
streams after using them to access the file?

> Le dim. 9 oct. 2022 à 15:23, Thomas Hoffmann (Speed4Trade GmbH)
>  a écrit :
>
> > Hello,
> > this might be a behavior of the underlying OS.
> > If the file is locked, it is marked for deletion and when the file lock is
> > released, the file is physically deleted.
> >
> > Maybe you can check with ProcessExplorer from MS whether there is an open
> > file handle on this file.
> >
> > Greetings,
> > Thomas
> >
> > > -Ursprüngliche Nachricht-
> > > Von: Martin Moore 
> > > Gesendet: Sonntag, 9. Oktober 2022 09:56
> > > An: Tomcat Users List ; ma...@apache.org
> > > Betreff: Re: About granting permissions to Tomcat JVM
> > >
> > > Hello Mark,
> > >
> > > I don't know if the SecurityManager is enabled or not (how to disable it
> > > btw?)
> > > I set the env var CATALINA_HOME to be C:/Program Files/Apache-Tomcat-8/
> > > The files in question are stored in Desktop/SomeFolder
> > >
> > > Thanks.
> > >
> > > Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a écrit :
> > >
> > > > On 08/10/2022 17:36, Martin Moore wrote:
> > > > > Hello,
> > > > >
> > > > > I am facing a problem using Tomcat V8 with my J2ee app that deletes
> > > > (using
> > > > > file.delete() Java 8) a file from disk (Windows). The file is
> > > > > actually deleting only on application level meaning that the
> > > > > application does not see the file anymore but if i open the folder i
> > > > > still see the file which
> > > > is
> > > > > then locked by Java process. I only get the file to be removed
> > > > > physically when i close the Tomcat instance.
> > > > >
> > > > > Does this problem relate to permissions in catalina.policy ?
> > > >
> > > > Unlikely.
> > > >
> > > > Are you using a SecurityManager?
> > > >
> > > > > How to solve this?
> > > >
> > > > Where, exactly, are you storing these files? Where, exactly, are
> > > > CATALINA_HOME and CATALINA_BASE?
> > > >
> > > > Mark
> > > >
> > > > -
> > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Question about upgrade from Tomcat 7 to Tomcat 9.0.67

[Suvendu Sekhar Mondal]

Hello Terry,

On Wed, Oct 5, 2022, 1:47 PM Terry ST SY/OGCIO 
wrote:

> Hi ,
>
> we would like to upgrade Tomcat 7 to Tomcat 9.0.67.
>
> Check on the major changes on Tomcat 7 to Tomcat 9. (One of the major
> change we initially spotted is the BIO connector used in Tomcat 7 for
> connector setup was removed in Tomcat 9:
> https://tomcat.apache.org/migration-9.html#BIO_connector_removed)
>
> For your reference, Http11Protocol was used in Tomcat 7, but since Tomcat
> 9 removed it, we tried to Http11NioProtocol with same parameters in the
> tomcat config in “conf/server.xml. Is it a proper method for migrate the
> BIO connector ( Tomcat 7 ) to NIO connector ( Tomcat 9 ) ?
>

That is the way to switch to NIO connector. You need to change the value of
protocol attribute of the connector tag like below:

protocol="org.apache.coyote.http11.Http11NioProtocol"

After that if you restart Tomcat, in startup log you'll see that NIO is
initialized. Also in thread dump thread name changes and becomes something
like "http-nio-*".


> Regards,
> Terry
>
>
>

Re: Tomcat unresponsive

[Suvendu Sekhar Mondal]

Hello Christoph,

On Thu, Sep 29, 2022, 3:27 PM Christoph Empl
 wrote:

> Hello,
>
> i’m facing a problem that my tomcat seems to become unresponsive if it’s
> under a certain load.
>

Can you please elaborate the situation a bit? What happens after that, is
it crashing or becomes sluggish? Does it recovers ever or you have to
restart it?

If you can recreate it at your will, can you please share load profile,
connector's configuration and some thread dumps? You may need to upload
dumps somewhere else and share the link. I will suggest to use jstack to
get simple thread dumps few seconds(say 5) apart.


> Most threads in threaddumps look like this:
>
>
> "https-jsse-nio-21030-exec-133" #13306 daemon prio=5 os_prio=0
> cpu=17099.37ms elapsed=897.51s tid=0x7f6270063b00 nid=0x1e5c38 in
> Object.wait()  [0x7f6109df4000]
>
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
> at java.lang.Object.wait(java.base@17.0.3/Native Method)
>
> - waiting on  available>
>
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1325)
>
> - locked <0x00050f48a720>
> (a java.util.concurrent.Semaphore)
>
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1226)
>
> at
> org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:805)
>
> at
> org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:42)
>
> at
> org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1185)
>
> at
> org.apache.coyote.http11.filters.IdentityInputFilter.end(IdentityInputFilter.java:151)
>
> at
> org.apache.coyote.http11.Http11InputBuffer.endRequest(Http11InputBuffer.java:655)
>
> at
> org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1192)
>
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:439)
>
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
>
> at org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
>
> at org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>
> - locked <0x00050f48a730>
> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
>
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
>
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
>
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>
> at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)
>
>
> I’m suspecting a slow network. Does anybody have another idea?
>
>
> Thanks, Christoph
>

Re: Connection pool

[Suvendu Sekhar Mondal]

Hello Mohamed,

On Thu, May 5, 2022 at 2:59 AM Mohamed Eliyas Abdul Kadar
 wrote:
>
> Hi All
> I am trying to limit the db connections on the oracle side by limiting the 
> size of connection pool in the datasource config as below. I tried setting 
> the max size to 20 by maxTotal/ maxActive either of them didn't work. The db 
> side connection was always more than 30. Please advise.
>
>
>   initialSize="10" maxTotal="10"  maxIdle="10"  maxWaitMillis="1"
> username="PORTAL" password="cde"
> driverClassName="oracle.jdbc.driver.OracleDriver"
> url="jdbc:oracle:thin:@cde" factory="SecureDataSource"/>
>

Can you please check the source IP of the connections just to make
sure they all originated from the server where you are running your
Tomcat? By any chance are you running multiple instances of Tomcats?

Thanks & Regards,
Suvendu

> Regards
> Mohamed
> This communication and its attachments contain confidential information and 
> is intended only for the named addressee. If you are not the named addressee 
> you should not disseminate, distribute or copy this communication. Please 
> notify the sender immediately if you have received this communication by 
> mistake and delete or destroy this communication. Communications cannot be 
> guaranteed to be secured or error-free as information could be intercepted, 
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
> The sender therefore does not accept liability for any errors or omissions in 
> the contents of this communication which arise as a result of transmission. 
> If verification is required please request a hard-copy version. NeoGenomics 
> Laboratories, 9490 NeoGenomics Way, Fort Myers, FL 33912, 
> http://www.neogenomics.com (2022)

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [ANN] ApacheCon NA 2022 in New Orleans, 3-6 Oct 2022, CFP is OPEN!

[Suvendu Sekhar Mondal]

Thank you Chris for the information!

On Thu, Apr 14, 2022 at 1:06 AM Christopher Schultz
 wrote:
>
> Suvendu,
>
> On 4/8/22 05:31, Suvendu Sekhar Mondal wrote:
> > Thank you Chris for the details and encouragement! :)
> >
> > One question, is there any plan to broadcast this year's ApacheCon as
> > well?  Personally speaking, I was excited to attend my first ApacheCon
> > when this even went virtual. Experience was overwhelming for me.
>
> This short answer is "probably not". Making the conference "+virtual" --
> even one-directional where the virtual participants can only watch+hear
> and not talk back -- would mean live-streaming everything which ends up
> being very expensive.
>
> In the past, we have had video + audio recordings of sessions, and those
> were usually not comprehensive: some sessions were audio-only, and some
> sessions had no recordings at all. That was expensive enough (which is
> why it wasn't comprehensive) without having to pay for the bandwidth to
> live-stream everything.
>
> So unless someone is willing to fund it and also organize it, it's
> probably not going to happen.
>
> Any sessions which /are/ recorded will be put up on the ASF's YouTube
> channel shortly after the conference has concluded.
>
> -chris
>
> > On Thu, Apr 7, 2022 at 7:57 PM Christopher Schultz
> >  wrote:
> >>
> >> All,
> >>
> >> [Cross-posting to dev@, please reply to users@]
> >>
> >> ApacheCon NA 2022 is back *in-person* in New Orleans, Louisiana. It will
> >> be held 3 - 6 October 2022 at the Canal Street Sheraton right next to
> >> the French Quarter.
> >>
> >> The call-for-presentations is currently open and we are looking to fill
> >> a 3-day track for Tomcat, so please submit your proposals today!
> >>
> >> https://www.apachecon.com/acna2022/
> >> (The link at the top is a little obscure, but at the top of the page
> >> there is a "Call for Presentations" where you can submit a proposal).
> >>
> >> Note that you don't have to have the presentation ready to go today in
> >> order to make a proposal. It's just gotta be ready around 30 seconds
> >> before you start to present it ;)
> >>
> >> (There is a tradition at ApacheCon of editing ones slides during the
> >> previous presentation. I don't recommend it, but anyone who is
> >> intimidated by the process can rest assured that even repeat-presenters
> >> are putting things together at the last moment.)
> >>
> >> Anyone who has never attended an ApacheCon should consider making this
> >> year their first: it's great fun, and you get to meet a lof of folks
> >> from all over the ASF, not just the Tomcat or httpd people, but folks
> >> working on projects you've never even heard of.
> >>
> >> If you aren't sure if you are interested in presenting, or aren't sure
> >> if you have the experience, knowledge, etc. to warrant a position as a
> >> speaker, please consider the following:
> >>
> >> 1. This is a welcoming community
> >> 2. This community exists to serve YOU
> >> 3. You are a part of this community
> >> 4. Helping others within the community encourages others to do the same
> >> 5. Topics can be very wide-ranging. Here are some examples of
> >> presentations from previous ApacheCon events:
> >>
> >> [From Committers / directly about Tomcat]
> >> - Running Apache Tomcat on GraalVM
> >> - Tomcat in clusters and clouds
> >> - Using Let's Encrypt with Tomcat
> >> - Securing Tomcat
> >> - Reverse-proxying Tomcat
> >> - Load balancing with Tomcat
> >> - Clustering with Tomcat
> >>
> >> [From Non-Committers or not directly about Tomcat]
> >> - Packaging Tomcat for Linux Distributions
> >> - I Love Lucee -- a Java implementation of Cold Fusion
> >> - Routing CDN traffic at scale using Tomcat
> >> - Secure Web Applications using Apache Fortress
> >> - Monitoring Tomcat; various tools
> >> - Building Reactive Applications on Tomcat
> >> - Troubleshooting performance using thread dumps
> >> - High Throughput Production Systems on Tomcat
> >> - Why I Love Open Source
> >> - Introduction to Spring Boot
> >> - Tomcat, TomEE, and Meecrowave
> >> - Apache Tomcat: Enabling Scripting Languages in JSPs
> >>
> >> If you are using Tomcat at $work and doing some

Re: How can I stop scanning TLD's?

[Suvendu Sekhar Mondal]

Hello Blake,

On Wed, Apr 13, 2022 at 6:34 PM Blake McBride  wrote:
>
> Thanks, Thomas.
>
> I tried putting the following in conf/context.xml
>
> 
>
>   
>
> 
> That worked.  However, that seems to match the Tomcat 8 docs and not the
> Tomcat 9 docs.  I will try figuring out the Tomcat 9 configuration as soon
> as I get a chance.
>
Just curious, is that not working in Tomcat 9?

There is another way to control the same thing. In
/conf/catalina.properties you can use following property to stop
scanning for TLDs:
tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar

But please remember, values of
tomcat.util.scan.StandardJarScanFilter.jarsToScan property will
override it.

> If you know the Tomcat 9 equivalent, that would be really helpful.
>
> Thanks!
>
> Blake
>
>
> On Wed, Apr 13, 2022 at 1:59 AM Thomas Hoffmann (Speed4Trade GmbH)
>  wrote:
>
> > > -Ursprüngliche Nachricht-
> > > Von: Blake McBride 
> > > Gesendet: Dienstag, 12. April 2022 22:31
> > > An: Tomcat Users List 
> > > Betreff: How can I stop scanning TLD's?
> > >
> > > Greetings,
> > >
> > > When booting my app, the system takes a long time to get past:
> > >
> > > 12-Apr-2022 20:21:18.648 INFO [main]
> > > org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was
> > scanned
> > > for TLDs yet contained no TLDs. Enable debug logging for this logger for
> > a
> > > complete list of JARs that were scanned but no TLDs were found in them.
> > > Skipping unneeded JARs during scanning can improve startup time and JSP
> > > compilation time.
> > > 12-Apr-2022 20:21:18.694 INFO [main]
> > > org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web
> > > application directory [/home/arahant/tomcat/webapps/host-manager] has
> > > finished in [277] ms
> > > 12-Apr-2022 20:21:18.695 INFO [main]
> > > org.apache.catalina.startup.HostConfig.deployDirectory Deploying web
> > > application directory [/home/arahant/tomcat/webapps/arahant]
> > > Apr 12, 2022 8:21:20 PM org.apache.jasper.servlet.TldScanner scanJars
> > > INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable
> > > debug logging for this logger for a complete list of JARs that were
> > scanned
> > > but no TLDs were found in them. Skipping unneeded JARs during scanning
> > > can improve startup time and JSP compilation time.
> > >
> > > I guess it is scanning for TLD's.  I'm not completely sure what that is
> > but if
> > > possible, I'd like to bypass that step as much as possible.
> > >
> > > Some facts:
> > >
> > > 64-bit Linux
> > > Apache Tomcat/9.0.62
> > > Java 8
> > >
> > > My app is pretty large - about 10,000 classes and over 100 jar files.  I
> > suppose
> > > scanning all of those files is what is taking so long.
> > >

Most of the cases JAR scanning takes long time but to find root cause
of slow Tomcat startup - take some thread dumps in few seconds
interval and analyze them later to find where thread is spending most
of the time OR attach a profiler to find hotspots.
> > > My app is dumb html and javascript files that communicate via SOAP and
> > > REST.  There is no JSP.
> > >
> > > Thanks for the help!
> > >
> > > Blake McBride
> >
> > Hello Blake,
> >
> > does this setting help out?
> > https://tomcat.apache.org/tomcat-9.0-doc/config/jar-scan-filter.html
> >
> > Greetings, Thomas
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >

Thanks & Regards,
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat service does not restart on Windows with high value of Xms/Xmx

[Suvendu Sekhar Mondal]

Hello Christophe,

On Tue, Apr 12, 2022 at 3:56 PM Morfin, Christophe  wrote:
>
> Hi
>
> As additional information I have done further testing
>
> Windows 2016 32Gb RAM
> Xms/Xmx 20g
> Pagefile 5g
>

Just wondering, how are you setting min and max heap now? By setting
Xms and Xmx values in the Java tab OR by setting values in the
"initial/max memory pool"? If you are using Xms and Xmx values, can
you remove it and try to set heap via "initial/max memory pool"
option? Just wanted to see if that makes any difference or not.

> Service restart works fine
>
> Same configuration on Windows 2019, restart fails with mentioned memory issue.
>
> Windows 2019 8gb RAM
> Xms/Xmx 7g
> Pagefile 2g
> Restart works fine
>
> So it seems to happen only on Windows 2019 with larger heap size.
>

This is indeed interesting. I don't have any Windows 2019 but I'll try with

> Note that this was reported to me by others, so it appears to be easily 
> reproducible .
> I wonder if you would have a chance to test on a Windows 2019 , 64 bit Build 
> 17763 (10.0.17763.2686) with above mentionned configuration as  I feel you 
> would reproduce.
>
> Thank you
> Kind regards
> Christophe
>
>
>
>
>
> -Original Message-
> From: Morfin, Christophe
> Sent: 11 April 2022 09:38
> To: 'users@tomcat.apache.org' 
> Subject: RE: Tomcat service does not restart on Windows with high value of 
> Xms/Xmx
>
> Hi Chris
>
> Thank you for the information, this is interesting indeed.
> Yes, I am using the default service installed by Tomcat installer so using 
> procrun and tomcat9.exe
>
> I have tried adding -DCATALINA_OPTS="-Xms20g -Xmx20g" under the Java Options 
> parameters in tomcat9w.exe > Java tab But this is not taken into account
>
> When I google this, it seems we can't set CATALINA_OPTS when starting as a 
> service
>
> Do you have further details on how to set CATALINA_OPTS when starting a 
> service ?
>
> Thank you
> Kind regards
> Christophe
>
>
>
>
> -Original Message-
> From: Morfin, Christophe
> Sent: 08 April 2022 17:57
> To: 'users@tomcat.apache.org' 
> Subject: RE: Tomcat service does not restart on Windows with high value of 
> Xms/Xmx
>
> Hi
>
> Thank you for your answers,
>
> Here below is the output of system part of hs_err file.
>
>
> Note though If I execute a manual stop and manual start immediately after, 
> this work, no issue.
> It is only when I use the Restart option.
>
> Both tests are done one after the other, so the overall amount of memory used 
> on the machine at the time is the same in both cases.
> When the service stops the committed memory goes down to 2.4 Gb, as nothing 
> else is running on that machine, so there is plenty memory left to restart 
> the process, which indeed happen when we use Stop then Start, but not with 
> Restart.
>
> I have spent some time with Microsoft too as I thought it was due to the OS, 
> but they are telling me it is not and it is related to Tomcat.
> I have a process dump that they help me collect. They mentioned it should be 
> helpful to you to check with the symbols.
> Let me know if you are interested about it.
>
> Thank you
> Christophe
>
>
>
> ---  S Y S T E M  ---
>
> OS: Windows Server 2019 , 64 bit Build 17763 (10.0.17763.2686) OS uptime: 0 
> days 8:03 hours Hyper-V role detected
>
> CPU:total 8 (initial active 8) (4 cores per cpu, 2 threads per core) family 6 
> model 85 stepping 7 microcode 0x, cmov, cx8, fxsr, mmx, sse, sse2, 
> sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 
> 3dnowpref, lzcnt, ht, tsc, bmi1, bmi2, adx, evex, fma
>
> Memory: 4k page, system-wide physical 32767M (30013M free) TotalPageFile size 
> 37631M (AvailPageFile size 14413M) current process WorkingSet (physical 
> memory assigned to process): 12M, peak: 12M current process commit charge 
> ("private bytes"): 61M, peak: 20541M
>
> vm_info: OpenJDK 64-Bit Server VM (11.0.14+9-LTS) for windows-amd64 JRE 
> (11.0.14+9-LTS), built on Jan 15 2022 01:18:46 by "Administrator" with 
> unknown MS VC++:1916
>
> END.
>
>
> Christophe Morfin
> Principal Technical Support Engineer
>
> 00800 782 4 4357 (00800 PTC 4 HELP)
> cmor...@ptc.com
>
> thingworx.com
>
>
> -Original Message-
> From: Morfin, Christophe
> Sent: 08 April 2022 15:10
> To: users@tomcat.apache.org
> Subject: Tomcat service does not restart on Windows with high value of Xms/Xmx
>
>
>
> Hi
>
> I am using Tomcat 9.0.52 and 9.0.60 on Windows server 2019 version 1809 (OS 
> Build 17763.2686) The machine has 32Gb of Ram Pagefile size 4Gb The Tomcat 
> service is configured with Xms20g Xmx20g added in the Java 8 parameters I am 
> using Amazon corretto 11.0.14
>
> Everything works fine if using Stop / Start in Control Panel > Services 
> However if the service is already started and I select Restart, then it stops 
> the service but fails to start with
>
> error 1067: The process terminated unexpectedly.
>
>
> Hs-err.pid file is created with OutOfMemoryError and:
>
> 

Re: Tomcat service does not restart on Windows with high value of Xms/Xmx

[Suvendu Sekhar Mondal]

Hello Christophe,

On Fri, Apr 8, 2022 at 7:40 PM Morfin, Christophe  wrote:
>
>
>
> Hi
>
> I am using Tomcat 9.0.52 and 9.0.60 on Windows server 2019 version 1809 (OS 
> Build 17763.2686)
> The machine has 32Gb of Ram
> Pagefile size 4Gb
> The Tomcat service is configured with Xms20g Xmx20g added in the Java 8 
> parameters
> I am using Amazon corretto 11.0.14
>
> Everything works fine if using Stop / Start in Control Panel > Services
> However if the service is already started and I select Restart, then it stops 
> the service but fails to start with
>
> error 1067: The process terminated unexpectedly.
>
>
> Hs-err.pid file is created with OutOfMemoryError and:
>
> There is insufficient memory for the Java Runtime Environment to continue.
> Native memory allocation (mmap) failed to map 21474836480 bytes for Failed to 
> commit area from 0x0003 to 0x0008 of length 
> 21474836480.
>

This error message indicates you are running out of Virtual
Memory(Physical Memory+Page file). When you are starting JVM with 20GB
heap, JVM will try to commit that amount. OS was not able to allocate
that much memory to JVM. How much memory was available at the time
when you restart Tomcat service?


> If setting Xms and Xmx to 16Gb, it works.
> If the pagefile size is increased to 32Gb, then it works also
Yes, this also confirms that. With 32GB page file now you have a total
of 64GB of Virtual Memory and the OS was able to happily allocate 20GB
to your JVM. With 16GB heap also you are asking less amount to
allocate.

> However we should not need that pagefile as when Tomcat is stopped, only 
> 2.5Gb out of the 32Gb are used, and it works just fine if we do the 2 step 
> process of Stop and then Start.
>
> Using other application, Zookeeper, does not show this problem, the restart 
> works ok even when set to 20Gb, so it seems something with the Tomcat service 
> management.
>

Since you mentioned it ONLY happens during restarts, I think some
other process(es) is/are consuming memory when JVM was running.

As you have a Java crash report(hs_err_pid*.log) available, can you
please post the output of the "system" section here? It'll tell us how
much memory was being used at that point. It can be found at the
bottom of that crash report.

> Any help will be appreciated
>
> Thank you
> Chris
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [ANN] ApacheCon NA 2022 in New Orleans, 3-6 Oct 2022, CFP is OPEN!

[Suvendu Sekhar Mondal]

Thank you Chris for the details and encouragement! :)

One question, is there any plan to broadcast this year's ApacheCon as
well?  Personally speaking, I was excited to attend my first ApacheCon
when this even went virtual. Experience was overwhelming for me.

Thanks & Regards,
Suvendu

On Thu, Apr 7, 2022 at 7:57 PM Christopher Schultz
 wrote:
>
> All,
>
> [Cross-posting to dev@, please reply to users@]
>
> ApacheCon NA 2022 is back *in-person* in New Orleans, Louisiana. It will
> be held 3 - 6 October 2022 at the Canal Street Sheraton right next to
> the French Quarter.
>
> The call-for-presentations is currently open and we are looking to fill
> a 3-day track for Tomcat, so please submit your proposals today!
>
> https://www.apachecon.com/acna2022/
> (The link at the top is a little obscure, but at the top of the page
> there is a "Call for Presentations" where you can submit a proposal).
>
> Note that you don't have to have the presentation ready to go today in
> order to make a proposal. It's just gotta be ready around 30 seconds
> before you start to present it ;)
>
> (There is a tradition at ApacheCon of editing ones slides during the
> previous presentation. I don't recommend it, but anyone who is
> intimidated by the process can rest assured that even repeat-presenters
> are putting things together at the last moment.)
>
> Anyone who has never attended an ApacheCon should consider making this
> year their first: it's great fun, and you get to meet a lof of folks
> from all over the ASF, not just the Tomcat or httpd people, but folks
> working on projects you've never even heard of.
>
> If you aren't sure if you are interested in presenting, or aren't sure
> if you have the experience, knowledge, etc. to warrant a position as a
> speaker, please consider the following:
>
> 1. This is a welcoming community
> 2. This community exists to serve YOU
> 3. You are a part of this community
> 4. Helping others within the community encourages others to do the same
> 5. Topics can be very wide-ranging. Here are some examples of
> presentations from previous ApacheCon events:
>
>[From Committers / directly about Tomcat]
>- Running Apache Tomcat on GraalVM
>- Tomcat in clusters and clouds
>- Using Let's Encrypt with Tomcat
>- Securing Tomcat
>- Reverse-proxying Tomcat
>- Load balancing with Tomcat
>- Clustering with Tomcat
>
>[From Non-Committers or not directly about Tomcat]
>- Packaging Tomcat for Linux Distributions
>- I Love Lucee -- a Java implementation of Cold Fusion
>- Routing CDN traffic at scale using Tomcat
>- Secure Web Applications using Apache Fortress
>- Monitoring Tomcat; various tools
>- Building Reactive Applications on Tomcat
>- Troubleshooting performance using thread dumps
>- High Throughput Production Systems on Tomcat
>- Why I Love Open Source
>- Introduction to Spring Boot
>- Tomcat, TomEE, and Meecrowave
>- Apache Tomcat: Enabling Scripting Languages in JSPs
>
>If you are using Tomcat at $work and doing something interesting,
> we'd love to hear about it.
>
> 6. You don't need to be the foremost expert in $feature to talk about it
> 7. We are actively looking for speakers to talk about these and other
> topics:
>
>- Deploying Tomcat in an auto-scaling environment (e.g. AWS EBS)
>- Tomcat should really have [Feature X]
>- Whatever you think might be interesting!
>
> Please consider speaking ESPECIALLY if you haven't done so before. If
> you are worried about whether your idea is good enough: don't. Just
> submit your idea to the CFP -- you don't have to write-up the
> presentation in order to submit an idea, just write a paragraph or two
> about what you want to do -- and the track chairpersons
> (chairpeople?[1]) will decide whether or not to include your
> presentation in the event. (And chances are good that if you submit an
> idea it will be accepted.)
>
> Please reply to the users list with any questions you may have about
> ApacheCon, the Tomcat track, or submitting a talk proposal.
>
> Thanks,
> -chris
>
> On behalf of all ApacheCon 2022 Tomcat Track chairpersons
>
>
> [1]
> https://vignette.wikia.nocookie.net/rickandmorty/images/c/cd/Furniture.png/revision/latest/scale-to-width-down/1000?cb=20160910223642
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: In what directory was tomcat installed

[Suvendu Sekhar Mondal]

Hello!

On Thu, Feb 24, 2022, 3:50 AM Amn Ojee Uw  wrote:

> I am trying to setup Tomecat in Netbeans, but after installing Tomcat in
> my Debian 11, and in Netbeans going to "Tools -> Servers->Add server
> ->Apache Tomcat or TomEE
> -> Server Location
> -> Browse"
>
> I cannot tell in which directory Tomcat 10.x was installed.
> I am not a qualified programmer, I am just someone who would like to
> program and to make things worst, I am  new to the Linux world. So... I
> am really lost here; any help will be much appreciated.
> Thanks in advance.
>

I have not used Netbeans for long times. Probably you can try to find where
CATALINA_HOME is pointing to. Normally that's the base of Tomcat in any
environment.

1. Start your Tomcat server from Netbeans
2. After that use ps and grep command to search for text "catalina". It'll
be something like:

ps aux | grep catalina

Thanks!
Suvendu

>

Re: JNDIRealm thread blocking issue

[Suvendu Sekhar Mondal]

On Fri, Oct 29, 2021 at 2:02 PM Rémy Maucherat  wrote:
>
> On Fri, Oct 29, 2021 at 9:28 AM Suvendu Sekhar Mondal  
> wrote:
> >
> > Hello Chris,
> >
> > On Fri, Oct 29, 2021 at 2:46 AM Christopher Schultz
> >  wrote:
> > >
> > > Suvendu,
> > >
> > > On 10/28/21 12:55, Suvendu Sekhar Mondal wrote:
> > > > Hello Everyone,
> > > >
> > > > I was investigating one thread pool exhaustion issue. Thread dump
> > > > analysis showed that all HTTP threads were waiting for a ReentrantLock
> > > > object. Object address 0x00066d727f28 were same for all of the
> > > > waiting threads:
> > > >
> > > > "http-nio-18100-exec-86" #32808 daemon prio=5 os_prio=0
> > > > tid=0x51835800 nid=0x29bc waiting on condition
> > > > [0x7a5be000]
> > > > java.lang.Thread.State: WAITING (parking)
> > > > at sun.misc.Unsafe.park(Native Method)
> > > > - parking to wait for  <0x00066d727f28> (a
> > > > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > > > at 
> > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > > > at 
> > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > > > at 
> > > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > > > at 
> > > > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> > > > at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > > > at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2385)
> > > > at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1274)
> > > >
> > > > There was no hint in the thread dump about which thread was owning the
> > > > lock. Luckily, one heap dump was taken before generating thread dump.
> > > > When I queried the heap dump for that ReentrantLock object, I saw that
> > > > another thread(http-nio-18100-exec-4) was holding the
> > > > lock(exclusiveOwnerThread). There was NO trace of
> > > > http-nio-18100-exec-4 thread in any of the thread dumps! So it was a
> > > > "lock without an owner" case.
> > >
> > > I think you are looking at several pieces of evidence that may or may
> > > not correlate to each other at all. The fact that the thread wasn't in
> > > the thread dump indicates that the thread (or even the whole JVM) had
> > > terminated between the time you took the heap-dump and the thread dump.
> > > Most likely, the monitor was owned by another thread when you took your
> > > thread-dump. Try using other tools which *do* reveal the lock-holders
> > > identity.
> > >
> >
> > This issue has happened a few times. "Busy Thread Count" was high
> > during the problem period. JVM was up and running when I collected
> > heap and thread dumps - pid was not changed in-between. I used jstack,
> > visualvm, jcmd - nothing revealed owing thread details. Only heap
> > dumps had some information on that object and which thread was holding
> > onto it. Here is a snap: https://pasteboard.co/D7dV3jej6zId.jpg
> >
> > I can simulate similar blocking without Tomcat with dummy code. There
> > also nothing reveals the owner's identity except the heap dump. Here
> > is sample: https://gist.github.com/suv3ndu/2ec9fe660d2b833996817ed62186eac2
> >
> > > > After glancing through the Tomcat’s JNDIRealm.get() code and
> > > > beyond[1], I can see lock is being acquired on singleConnectionLock.
> > > > That lock is getting released either in the close() or release()
> > > > method. So, if something bad happens to the thread which is trying to
> > > > establish a connection, then lock will be held without a proper owner
> > > > and a thread blocking situation will be created. Am I interpreting the
> > > > code correctly? Should we not handle any failure inside get()?
> > > >
> > > > Also, I still have not got the reason why the thread got terminated.
> > > > Any suggestions on how I can enable any specific logging?
> > > >
> > > > My setup is:
> > > > Tomcat version: 9.0.39
> > > > Connector: NIO
>

Re: JNDIRealm thread blocking issue

[Suvendu Sekhar Mondal]

Hello Chris,

On Fri, Oct 29, 2021 at 2:46 AM Christopher Schultz
 wrote:
>
> Suvendu,
>
> On 10/28/21 12:55, Suvendu Sekhar Mondal wrote:
> > Hello Everyone,
> >
> > I was investigating one thread pool exhaustion issue. Thread dump
> > analysis showed that all HTTP threads were waiting for a ReentrantLock
> > object. Object address 0x00066d727f28 were same for all of the
> > waiting threads:
> >
> > "http-nio-18100-exec-86" #32808 daemon prio=5 os_prio=0
> > tid=0x51835800 nid=0x29bc waiting on condition
> > [0x7a5be000]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0x00066d727f28> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > at 
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > at 
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > at 
> > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > at 
> > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> > at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2385)
> > at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1274)
> >
> > There was no hint in the thread dump about which thread was owning the
> > lock. Luckily, one heap dump was taken before generating thread dump.
> > When I queried the heap dump for that ReentrantLock object, I saw that
> > another thread(http-nio-18100-exec-4) was holding the
> > lock(exclusiveOwnerThread). There was NO trace of
> > http-nio-18100-exec-4 thread in any of the thread dumps! So it was a
> > "lock without an owner" case.
>
> I think you are looking at several pieces of evidence that may or may
> not correlate to each other at all. The fact that the thread wasn't in
> the thread dump indicates that the thread (or even the whole JVM) had
> terminated between the time you took the heap-dump and the thread dump.
> Most likely, the monitor was owned by another thread when you took your
> thread-dump. Try using other tools which *do* reveal the lock-holders
> identity.
>

This issue has happened a few times. "Busy Thread Count" was high
during the problem period. JVM was up and running when I collected
heap and thread dumps - pid was not changed in-between. I used jstack,
visualvm, jcmd - nothing revealed owing thread details. Only heap
dumps had some information on that object and which thread was holding
onto it. Here is a snap: https://pasteboard.co/D7dV3jej6zId.jpg

I can simulate similar blocking without Tomcat with dummy code. There
also nothing reveals the owner's identity except the heap dump. Here
is sample: https://gist.github.com/suv3ndu/2ec9fe660d2b833996817ed62186eac2

> > After glancing through the Tomcat’s JNDIRealm.get() code and
> > beyond[1], I can see lock is being acquired on singleConnectionLock.
> > That lock is getting released either in the close() or release()
> > method. So, if something bad happens to the thread which is trying to
> > establish a connection, then lock will be held without a proper owner
> > and a thread blocking situation will be created. Am I interpreting the
> > code correctly? Should we not handle any failure inside get()?
> >
> > Also, I still have not got the reason why the thread got terminated.
> > Any suggestions on how I can enable any specific logging?
> >
> > My setup is:
> > Tomcat version: 9.0.39
> > Connector: NIO
> > JDK: AdoptOpenJDK: 1.8.192
> > OS: Windows 2016
>
> Looks like you need a whole bunch of upgrades. Search the Tomcat 9.x
> changelog for "JNDIRealm" and you'll see there have been changes since
> 9.0.39 that may have already resolved this issue. Are you able to
> re-test with Tomcat 9.0.54?
>

It will not be easy for me to upgrade it and test it. Lots of approval
is required to get that done. :(

>  > [1]
> https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/catalina/realm/JNDIRealm.java#L2553
>
> Note that you are looking at the current version of JNDIRealm.java. The
> version you are running is 17 commits behind that.
>
> The line of code calling ReentrantLock.lock in your code would be
> https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/ca

JNDIRealm thread blocking issue

[Suvendu Sekhar Mondal]

Hello Everyone,

I was investigating one thread pool exhaustion issue. Thread dump
analysis showed that all HTTP threads were waiting for a ReentrantLock
object. Object address 0x00066d727f28 were same for all of the
waiting threads:

"http-nio-18100-exec-86" #32808 daemon prio=5 os_prio=0
tid=0x51835800 nid=0x29bc waiting on condition
[0x7a5be000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00066d727f28> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at 
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.apache.catalina.realm.JNDIRealm.get(JNDIRealm.java:2385)
at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1274)

There was no hint in the thread dump about which thread was owning the
lock. Luckily, one heap dump was taken before generating thread dump.
When I queried the heap dump for that ReentrantLock object, I saw that
another thread(http-nio-18100-exec-4) was holding the
lock(exclusiveOwnerThread). There was NO trace of
http-nio-18100-exec-4 thread in any of the thread dumps! So it was a
"lock without an owner" case.

After glancing through the Tomcat’s JNDIRealm.get() code and
beyond[1], I can see lock is being acquired on singleConnectionLock.
That lock is getting released either in the close() or release()
method. So, if something bad happens to the thread which is trying to
establish a connection, then lock will be held without a proper owner
and a thread blocking situation will be created. Am I interpreting the
code correctly? Should we not handle any failure inside get()?

Also, I still have not got the reason why the thread got terminated.
Any suggestions on how I can enable any specific logging?

My setup is:
Tomcat version: 9.0.39
Connector: NIO
JDK: AdoptOpenJDK: 1.8.192
OS: Windows 2016

[1] 
https://github.com/apache/tomcat/blob/57a6a40fc9f995e4d449358bbde047aab6d9f39a/java/org/apache/catalina/realm/JNDIRealm.java#L2553

Thanks & Regards,
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [OT] visualvm time stamps

[Suvendu Sekhar Mondal]

Hi Chris,

On Sat, Apr 10, 2021 at 12:33 AM Chris Cheshire  wrote:
>
> My googlefu is failing me here.
>
> I am trying to figure out some anomalous database connection behavior in my 
> tomcat web app. I have enabled JMX/RMI and have visualvm running on my local 
> machine.
>
> I found the ability to monitor the active connections as a live chart, and it 
> has an export data function. This export creates a csv with what is supposed 
> to be a time stamp and a count but the time stamp is in a 5.6 format. I have 
> never seen this before. How do I convert this into something normal - millis 
> since epoch or even a human readable ISO format?
>
> Example
> 44295.607552
>

As far as I know, out of the box visualvm(I have 2.0.2) do not have
any option to export CPU/GC/Heap details. Are you using any plugins to
export data?

> Chris
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Not able to connect to Tomcat 9.0.39 instance using jconsole/jvisualvm

[Suvendu Sekhar Mondal]

On Thu, Feb 4, 2021 at 2:26 PM Mark Thomas  wrote:
>
> On 04/02/2021 08:08, Luis Rodríguez Fernández wrote:
> > Hello Suvendu,
> >
> > I've never used the
> > "org.apache.catalina.mbeans.JmxRemoteLifecycleListener", I would advise you
> > to continue using the JVM startup options for JMX [1]
>
> +1. Ignore the JmxRemoteLifecycleListener and use the settings provided
> by the JRE.
>
> Mark
>

Thank you guys!
>
> > Martynas: the JPDA port is using to enable debugging in your java process
> > and be able to connect to it, e.g. via your favourite IDE.
> >
> > Cheers,
> >
> > Luis
> >
> > [1]
> > https://tomcat.apache.org/tomcat-9.0-doc/monitoring.html#Enabling_JMX_Remote
> >
> >
> >
> >
> >
> >
> > El mar, 2 feb 2021 a las 16:23, Suvendu Sekhar Mondal ()
> > escribió:
> >
> >> Hi Martynas,
> >>
> >> On Tue, Feb 2, 2021 at 5:04 PM Martynas Jusevičius
> >>  wrote:
> >>>
> >>> Not sure if related, but JPDA address config changed from -
> >>> JPDA_ADDRESS=8000 on Tomcat 8 to - JPDA_ADDRESS=*:8000 on Tomcat 9
> >>> (i.e. host needs to be included, or a wildcard).
> >>>
> >> Thanks for pointing that out but I think it is not related to the
> >> problem I am seeing.
> >>
> >>> On Tue, Feb 2, 2021 at 12:22 PM Suvendu Sekhar Mondal 
> >> wrote:
> >>>>
> >>>> Hello Everyone,
> >>>>
> >>>> We recently migrated Tomcat from 7.0.55 to 9.0.39. Everything is
> >>>> working as expected except accessing exposed MBeans via JMX clients
> >>>> like jconsole/jvisualvm. While troubleshooting the issue, I enabled
> >>>> debug logging for both of those tools and it is throwing following
> >>>> error:
> >>>> java.rmi.ConnectIOException: non-JRMP server at remote endpoint
> >>>> at
> >> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248)
> >>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> >>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
> >>>> at
> >> sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
> >>>> at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
> >>>> at sun.tools.jconsole.ProxyClient.(ProxyClient.java:127)
> >>>> at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:475)
> >>>> at sun.tools.jconsole.JConsole$3.run(JConsole.java:524)
> >>>>
> >>>> We are using org.apache.catalina.mbeans.JmxRemoteLifecycleListener to
> >>>> specify RMI registry and server port like this:
> >>>>>>>> className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
> >>>> rmiRegistryPortPlatform="8181" rmiServerPortPlatform="8282" />
> >>>>
> >>>> And we can see that TC is registering properly to those ports:
> >>>>  2021-02-02 05:07:08,541 INFO
> >>>> org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The JMX Remote
> >>>> Listener has configured the registry on port [8181] and the server on
> >>>> port [8282] for the [Platform] server
> >>>>
> >>>> We use remote JMX with no authentication or SSL:
> >>>> -Dcom.sun.management.jmxremote.ssl=false
> >>>> -Dcom.sun.management.jmxremote.authenticate=false
> >>>>
> >>>> Workaround is to add following options in JVM arguments and then I was
> >>>> able to use JMX on port 8181:
> >>>> -Dcom.sun.management.jmxremote
> >>>> -Dcom.sun.management.jmxremote.port=8181
> >>>>
> >>>> But I am not sure why it broke in Tomcat 9.0.39 in the first place
> >>>> because with a similar configuration we are able to access JMX on
> >>>> Tomcat 7.0.55. I noticed that JmxRemoteLifecycleListener has been
> >>>> deprecated and will be removed in future[1] but we are on a version
> >>>> which was released 3-4 months ago. So, could this be a bug or
> >>>> something else?
> >>>>
> >>>> [1] 2021-02-02 05:07:07,447 WARNING
> >>>> org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The
> >>>> JmxRemoteLifecycleListener is deprecated as as the features it
> >&g

Re: Not able to connect to Tomcat 9.0.39 instance using jconsole/jvisualvm

[Suvendu Sekhar Mondal]

Hi Martynas,

On Tue, Feb 2, 2021 at 5:04 PM Martynas Jusevičius
 wrote:
>
> Not sure if related, but JPDA address config changed from -
> JPDA_ADDRESS=8000 on Tomcat 8 to - JPDA_ADDRESS=*:8000 on Tomcat 9
> (i.e. host needs to be included, or a wildcard).
>
Thanks for pointing that out but I think it is not related to the
problem I am seeing.

> On Tue, Feb 2, 2021 at 12:22 PM Suvendu Sekhar Mondal  
> wrote:
> >
> > Hello Everyone,
> >
> > We recently migrated Tomcat from 7.0.55 to 9.0.39. Everything is
> > working as expected except accessing exposed MBeans via JMX clients
> > like jconsole/jvisualvm. While troubleshooting the issue, I enabled
> > debug logging for both of those tools and it is throwing following
> > error:
> > java.rmi.ConnectIOException: non-JRMP server at remote endpoint
> > at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248)
> > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
> > at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
> > at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
> > at sun.tools.jconsole.ProxyClient.(ProxyClient.java:127)
> > at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:475)
> > at sun.tools.jconsole.JConsole$3.run(JConsole.java:524)
> >
> > We are using org.apache.catalina.mbeans.JmxRemoteLifecycleListener to
> > specify RMI registry and server port like this:
> >> className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
> > rmiRegistryPortPlatform="8181" rmiServerPortPlatform="8282" />
> >
> > And we can see that TC is registering properly to those ports:
> >  2021-02-02 05:07:08,541 INFO
> > org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The JMX Remote
> > Listener has configured the registry on port [8181] and the server on
> > port [8282] for the [Platform] server
> >
> > We use remote JMX with no authentication or SSL:
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> >
> > Workaround is to add following options in JVM arguments and then I was
> > able to use JMX on port 8181:
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.port=8181
> >
> > But I am not sure why it broke in Tomcat 9.0.39 in the first place
> > because with a similar configuration we are able to access JMX on
> > Tomcat 7.0.55. I noticed that JmxRemoteLifecycleListener has been
> > deprecated and will be removed in future[1] but we are on a version
> > which was released 3-4 months ago. So, could this be a bug or
> > something else?
> >
> > [1] 2021-02-02 05:07:07,447 WARNING
> > org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The
> > JmxRemoteLifecycleListener is deprecated as as the features it
> > provides are now available in the remote JMX capability included with
> > the JRE. This listener will be removed in Tomcat 10 and may be removed
> > from Tomcat 9 some time after 2020-12-31.
> >
> > JDK version: jdk1.8.0_192
> > OS: Windows Server 2016
> >
> > Thanks & Regards,
> > Suvendu
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Not able to connect to Tomcat 9.0.39 instance using jconsole/jvisualvm

[Suvendu Sekhar Mondal]

Hello Everyone,

We recently migrated Tomcat from 7.0.55 to 9.0.39. Everything is
working as expected except accessing exposed MBeans via JMX clients
like jconsole/jvisualvm. While troubleshooting the issue, I enabled
debug logging for both of those tools and it is throwing following
error:
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:248)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112)
at sun.tools.jconsole.ProxyClient.checkSslConfig(ProxyClient.java:234)
at sun.tools.jconsole.ProxyClient.(ProxyClient.java:127)
at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:475)
at sun.tools.jconsole.JConsole$3.run(JConsole.java:524)

We are using org.apache.catalina.mbeans.JmxRemoteLifecycleListener to
specify RMI registry and server port like this:
  

And we can see that TC is registering properly to those ports:
 2021-02-02 05:07:08,541 INFO
org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The JMX Remote
Listener has configured the registry on port [8181] and the server on
port [8282] for the [Platform] server

We use remote JMX with no authentication or SSL:
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Workaround is to add following options in JVM arguments and then I was
able to use JMX on port 8181:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8181

But I am not sure why it broke in Tomcat 9.0.39 in the first place
because with a similar configuration we are able to access JMX on
Tomcat 7.0.55. I noticed that JmxRemoteLifecycleListener has been
deprecated and will be removed in future[1] but we are on a version
which was released 3-4 months ago. So, could this be a bug or
something else?

[1] 2021-02-02 05:07:07,447 WARNING
org.apache.catalina.mbeans.JmxRemoteLifecycleListener - The
JmxRemoteLifecycleListener is deprecated as as the features it
provides are now available in the remote JMX capability included with
the JRE. This listener will be removed in Tomcat 10 and may be removed
from Tomcat 9 some time after 2020-12-31.

JDK version: jdk1.8.0_192
OS: Windows Server 2016

Thanks & Regards,
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: HTTP thread pool size remained high after database failover

[Suvendu Sekhar Mondal]

Thank you Chris for your response.

On Wed, Nov 11, 2020 at 10:02 PM Christopher Schultz
 wrote:
>
> Suvendu,
>
> On 11/11/20 06:40, Suvendu Sekhar Mondal wrote:
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 
> > 2016
>
> As Michael pointed-out, these all need to be upgraded, but they are not
> affecting your experience, here.
>
> > Observations:
> > After database failover, we noticed that HTTP thread pool size jumped
> > from 10 to 50. No. of busy threads jumped to 40 for a minute and then
> > came down to normal range(1-5). But thread pool size remained 50 for
> > the next one hour. When we collected the heap dump after one hour,
> > thread pool size jumped again to ~90. Thread dump analysis is showing
> > most of the HTTP threads were just waiting to get some work[1]. That
> > looked surprising to me. Few of them are runnable and doing some work
> > - which matches "busy thread" count. During all this timeframe load on
> > the system remained the same.
> >
> > We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
> > do not have any keepAliveTimeout set in Tomcat. App uses session
> > affinity. We were looking for some connector properties which deal
> > with idle threads but found none[3].
>
> You won't find them.
>
Okay.
> > We are not seeing any "problem" in our app after DB failover but this
> > is something also newer to us and we are trying to explain it.
>
> What are you trying to explain?
>
We are simply trying to understand this behavior as this is something
new we noticed.
> > Tomcat's memory usage increased by a few MB as new threads were
> > spawned. My expectation was that if threads in the thread pool are not
> > getting used and idle, after sometime thread pool size will reduce to
> > "minSpareThreads" size which is 10 in our case. Is that expectation
> > wrong? Can you please share your thoughts?
>
> This is a common misconception. If you do not configure an "executor"
> then Tomcat will create one for you and it will NOT remove idle threads
> from the thread pool.
>
> The solution is to manually-specify an Executor:
>
>  minSpareThreads="Y" />
>
> 
>
> With this configuration, you should see your thread count stay between Y
> and X, but when X are not necessary, your thread count should drop
> towards Y as appropriate.
>

Thanks for the explanation. This is the kind of information I was
looking for in Tomcat docs but was not able to locate it. We will try
this approach and let you know the outcome.
> Hope that helps,
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: HTTP thread pool size remained high after database failover

[Suvendu Sekhar Mondal]

Hi Michael,

On Wed, Nov 11, 2020 at 6:03 PM Michael Osipov  wrote:
>
> Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:
> > Hello Everyone,
> >
> > During database failover resiliency testing we noticed something
> > unusual which we still cannot explain. So, reaching out to the
> > community for help.
> >
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 
> > 2016
>
> Almost of them them are outdated, you should upgrade first.

I know. We will be upgrading them in the next release. For this
release we have to use aforementioned softwares.

For the last three releases we used the same softwares but had not
noticed this issue - maybe we were not looking that closely. Now, we
are seeing this and can't let it go without explaining it. :)

So, any thoughts on this?

>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

HTTP thread pool size remained high after database failover

[Suvendu Sekhar Mondal]

Hello Everyone,

During database failover resiliency testing we noticed something
unusual which we still cannot explain. So, reaching out to the
community for help.

Application Setup:
AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016

Observations:
After database failover, we noticed that HTTP thread pool size jumped
from 10 to 50. No. of busy threads jumped to 40 for a minute and then
came down to normal range(1-5). But thread pool size remained 50 for
the next one hour. When we collected the heap dump after one hour,
thread pool size jumped again to ~90. Thread dump analysis is showing
most of the HTTP threads were just waiting to get some work[1]. That
looked surprising to me. Few of them are runnable and doing some work
- which matches "busy thread" count. During all this timeframe load on
the system remained the same.

We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
do not have any keepAliveTimeout set in Tomcat. App uses session
affinity. We were looking for some connector properties which deal
with idle threads but found none[3].

We are not seeing any "problem" in our app after DB failover but this
is something also newer to us and we are trying to explain it.
Tomcat's memory usage increased by a few MB as new threads were
spawned. My expectation was that if threads in the thread pool are not
getting used and idle, after sometime thread pool size will reduce to
"minSpareThreads" size which is 10 in our case. Is that expectation
wrong? Can you please share your thoughts?

[1]
"http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0
tid=0x45f03000 nid=0x30d4 waiting on condition
[0x5a4cf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0006a5400178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

[2]




[3]
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html

Thanks & Regards,
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Logging Information when Tomcat reaches max thread limit

[Suvendu Sekhar Mondal]

Hello Aquib,

On Mon, Nov 2, 2020 at 10:09 AM Aquib Khan  wrote:
>
> Hi,
>
> We have a usecase where we want that our application should indicate when it 
> reaches maximum thread limit. Our application is deployed in tomcat. Maximum 
> thread limit of tomcat is 200,  so If 200 threads are reached, does tomcat 
> provide any logger information?
>

As far as I know, there is no logging which will tell you that the
thread pool got exhausted but I am pretty sure your application will
react to it by throwing connection related exception as you mentioned
below.
> We tried making multiple rest calls and reducing max threads to 2-3. We faced 
> ConnectionRefused and socketTimeout Exceptions on client side. But no 
> exception or logger in Catalina logs. We changed Logging level as well in 
> logging.properties.
>
> We tried Changing values of maxThreads, minspareThread, acceptCount in 
> server.xml. We used Jmeter for making rest calls.
>
> Can we get any help here? Does tomcat provides any information when it 
> reaches max thread limit? Do we have to change any property in server.xml or 
> in logging.properties
>

Tomcat exposes different thread pool properties via JMX which are
available under JMX Object Name:"Catalina:type=ThreadPool". I think in
your case "currentThreadCount" and "currentThreadsBusy" will be
helpful. "currentThreadCount" will give you thread pool size.
"currentThreadsBusy" will tell you how many threads are busy
processing requests. Normally if the "currentThreadsBusy" count goes
up then it indicates either you have thread blocking or lots of
requests are coming in. Based on those numbers, you need to decide on
the thread pool size. There are more properties and operations exposed
via JMX. You can explore all of them by using jconsole or JMC. If you
have any Application Performance Management(APM) tool like
AppDynamics, Dynatrace, New Relic etc. it will make your life a bit
easier. Those tools usually can throw alerts if thread pool
utilization is high and based on which you can act.

> Any help in this regard would be highly appreciable.
> Thanks in Advance.
>
> Regards,
> Aquib

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [OT] Trying to determine the minimum heap required for an operation

[Suvendu Sekhar Mondal]

Hello Christopher,

On Mon, Jul 6, 2020, 8:50 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> All,
>
> Definitely off-topic, but it's the kind of weird thing someone here
> might have experience with.
>
> I have an offline operation I'm considering bringing "inside" my
> web-based application. My only concern is memory usage: it requires
> that a bunch of data be loaded from a db into memory and then
> analyzed. It doesn't take long to execute -- maybe 10 seconds or so,
> so the memory can be released back to the rest of the application.
>
> I've instrumented the command-line process with a thread which runs
> every .5sec and captures the used-memory, maintaining a high-water
> mark and reporting it after the whole operation is done. The first
> time I ran it (with no specific JVM memory-related settings), it
> reported that the high-water mark was ~450MiB.
>
> I figured that was higher than necessary, and probably just
> represented a lazy GC with loads of memory, so I constrained the
> process using -Xmx64M. That resulted in a 16MiB high-water mark. I
> tried again with -Xmx8M and the high-water mark became 5MiB.
>
> Is there a particularly good way to force the GC to be as aggressive
> as possible to see how low I can go, or should I just keep
> playing-around with the -Xmx setting.
>

Looks like you wanted to track live objects generated by your program. I
don't recall a JMX metrics of JVM to get that. If someway you can trigger
couple of Full GCs during processing and logging everything on a GC log,
then you'll get that. So, reducing -Xmx will help after some iterations.

One thing you can try using is -Dsun.rmi.dgc.client.gcInterval and server
flags and set them to 1 sec. That way you will get Full GC every 1 sec. It
worked properly with Parallel GC and doesn't work with G1 in JDK 1.8. You
can try and see if serves your purpose.

More on that property can be found at
https://docs.oracle.com/javase/8/docs/technotes/guides/rmi/sunrmiproperties.html

>
> Another option is to serialize my in-memory structure to the disk to
> get a sense of the size in-memory, though it's really not the same --
> it will at least get me in the right ballpark.
>
> Any suggestions?
>
> Thanks,
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP6U8
> pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jxF
> dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu4x
> Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+sCE
> ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3BuY
> rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh
> 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5
> x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ/SC
> uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ
> 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj
> 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY
> F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88=
> =u12k
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: JVM job for Tomcat taking lots and lots of CPU

[Suvendu Sekhar Mondal]

Hello James,

On Wed, Feb 12, 2020, 6:10 AM James H. H. Lampert 
wrote:

> Ladies and Gentlemen:
>
> We have a customer installation in which the JVM job for our Tomcat
> server is frequently using massive amounts of CPU.
>
> It's Tomcat 7.0.67, running on an AS/400, in a 64-bit Java 7 JVM, with
> -Xms3096m and -Xmx5120m JVM arguments.
>

Is this a snippet from GC log or collected data?

>
> GC information on the JVM job shows:
> > Garbage collected heap:
> >   Initial heap size  . . . . . . . . . :  3096.000M
> >   Maximum heap size  . . . . . . . . . :  5120.000M
> >   Current heap size  . . . . . . . . . :  4458.562M
> >   Heap in use  . . . . . . . . . . . . :  1907.673M
>

If I am reading it correctly, actual heap usage was lower than the initial
heap size. JVM still got ~3GB space left for it's use. I am not seeing any
reason for JVM to do frequent GC unless heap is fragmented and unusable to
JVM.

> Other memory:
> >   Internal (break) memory size . . . . :   504.982M
> >   JIT memory size  . . . . . . . . . . :74.000M
> >   Shared classes memory size . . . . . : 0.000M
> > General GC information:
> >   Current GC cycle . . . . . . . . . . :   2184
> >   GC policy type . . . . . . . . . . . : GENCON
>

Is this J9 JVM? Hotspot vms do not have that GC policy.

>   Current GC cycle time  . . . . . . . :552

>   Accumulated GC time  . . . . . . . . :5108241
>

Are those values are in Millisecond or second? If in MS, on avg 2.3
sec(5108241/2184)
is the collection time, which is bit high. But I don't know anything about
the application. Also I don't know if there were sudden spikes which might
have skewed the avg.

>
> It seems to be doing a lot of garbage-collecting.
>
> Would switching to Java 8 help? Would switching to 7.0.93 help?
>

I will suggest to analyze the garbage collection logs thoroughly before
making such decisions. You can share it with us. Upgrade to Java 8 has some
effort as Permgen was removed and Metaspace was introduced which now runs
on native memory. So some testing needs to be done to make sure app is not
leaking in Metaspace.

>
> --
> James H. H. Lampert
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Windows: Tomcat grabs 100% CPU after upgrading to 7.0.96

[Suvendu Sekhar Mondal]

On Tue, Aug 13, 2019, 11:24 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Paula,
>
> On 8/13/19 13:11, Childers, Paula wrote:
> > Windows Server 2008r2,  Oracle Java 1.7.0_231
>
> Thanks for that.
>
> > We are running multiple (6) Tomcats (each as a service) on this
> > server.  Recently installed 7.0.96 for all six, as an upgrade from
> > 7.0.94.
> >
> > We are seeing at least one of the Tomcats will attempt to grab as
> > much CPU as possible, until the system is at 100%.> Tried different
> > combinations of the different Tomcats running to see if the problem
> > was with a specific one, but the behavior occurs in multiple
> > combinations.  As long as there is more than one Tomcat running,
> > one of them will try to grab 100% CPU.  (Almost like it is in a
> > loop? Or competing for a resource?)
> >
> > If I start up only one Tomcat, it runs normally.
>
> Are you using 6 different installations, or a split-installation with
> CATALINA_HOME and different CATALINA_BASES? It shouldn't really
> matter, but it would be good to know.
>
> > Bumped logging to DEBUG, nothing additional showed up, all logs
> > look normal.
> >
> > Shut down all the 7.0.96 Tomcats, brought the previous 7.0.94
> > Tomcats back up, and all of them run normally, no CPU hogging -
> > total CPU consumption fluctuates around 15% unless there is heavy
> > usage.
> >
> > None of the web applications have changed anytime recently.
> >
> > Java was updated from 1.7.0_221 to 1.7.0_231 at the same time as
> > the Tomcat 7.0.96 upgrade, but the 7.0.94 Tomcats do not seem to
> > have any problems with the newer Java version.
> That shouldn't really affect much unless there happens to be a really
> bad JVM bug. With a version number like that, I wouldn't expect a big
> bug like that to be present. You might want to upgrade to an even more
> recent version of Java. The world is pretty much on Java 12 at this
> point. (I say this personally as a Java 8 hold-out.)
>
> > When we initially upgraded to 7.0.96, there were some issues with
> > accessing certain files and disk areas due to the "default" user
> > change to "local service" - we changed the user back to the
> > previous "Local System" to allow time for examining the permissions
> > changes that will be necessary to use "local service" in our
> > environment.
>
> That should be "safe" as far as Tomcat is concerned. You may want to
> investigate a little more about how best to set permissions for your
> Tomcat user.
>
> Can you take some thread dumps of the 100%-CPU-process a few seconds
> apart to see what the JVM is doing? Feel free to post the whole thread
> dump to the list if you aren't quite sure how to read it.
>

In addition to Chris's suggestion, please use ProcessExplorer on the rouge
Tomcat process and identify the thread(s) which are consuming CPU. Once
identified, you can correlate threads from ProcessExplorer data with thread
dumps and determine what they are doing.


> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl1S+TYACgkQHPApP6U8
> pFja9xAAmn0BZIIIQXMB5T5Kc0CNqY3m5/qjlyZjnXzvJCRPmTp9hksKEoy0Upbd
> u/aSii/Z9YpTVD6wrltv5yafncdeUcsJR7Xc1XBu59h9vo9KsSxL9x7zUEH0NJ64
> KAHBgpqm8VKiXBsngi2LG57eUNAmop/FA8mmkiJm2pemcBb/Kf2HMZlpDVmpY+MA
> sugWb/5/g3GbQvF9azXVZZ9LP0Ot4S7tBJH2w/8K/sEi+IXfAtRcYFivMtZo2G3l
> 7aXJ8oDDILlBoo9BMXAY1T0VJpB/VzQr5xkRjzgLiUDY7vXP9u7bKW6io+9x2Kxu
> kqrchYiOcsLLddQqtpWEDdfgqplUm7tUxVtfD0YIdu50B/KFHQtdsxUsTIoWSxOq
> bES9R2Z756/d/sdeh6g/o+1Svkp6vrKlMwfNhBl+9ZMHENhJMUr+fqdHlcMlFZF5
> 3AaQYOc/azgs0NkSuVkb2c3ThZXhAV7VQEbLPjeOWZFocJoZQCrbn5u0VwFAgFDo
> 2nYW/rA5SUBofoZCL6ejPg9WELIFVDUgdYeK2jc+TvQ5XJIU34X1Mnw6AkU3jKtp
> Brch/ei8/uks45nBOMrDMiBWnvtgKSc+6kN+ak/juGsSjAp32YFBDOBSNjfaSDIm
> iYInyuay9Cpto5lR5daR/fUK1OX6rANSamoFqYHt07y48Yj+VQQ=
> =yfbP
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: OT: Connection timeout with HttpClient

[Suvendu Sekhar Mondal]

Hi Chris,

On Tue, Jul 23, 2019 at 6:00 PM Christopher Schultz
 wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Suvendu,
>
> On 7/23/19 07:39, Suvendu Sekhar Mondal wrote:
> > One of our legacy applications is using Apache Commons HttpClient
> > 3.1. POST call to one REST service is failing with
> > "java.net.ConnectException: Connection timed out: connect"
> > exception[1]. Timeout is occurring after one minute. To figure out
> > where thread is spending all the time, I took multiple thread
> > dumps. In all of them, thread is trying to create secure socket[2].
> > I am not seeing any thread pool issue. Also the target REST service
> > is working properly with same payload while invoked from Postman.
> > It is only failing consistently when going through the HttpClient.
> > I am trying to figure out why it is taking so long in connecting to
> > the socket. I am looking for suggestion on how to attack this one.
> >
> > Tomcat: 7.0.55 JRE: 1.8_92 -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
>
> Your version is Tomcat is quite old and contains numerous "important"
> publicly-known vulnerabilities. You should upgrade ASAP.
>
Yes, I agree. There are some "technical debt" and upgrade Tomcat and
Apache are one of them. We have plans to do it on next release. :)

> The connection timeout you are seeing is on the client end: your
> client is connecting to another server and the server isn't responding
> fast enough. That's why your stack trace always shows the thread
> "creating a socket": it's trying to connect to the remote service and
> it's never completing.
>
> Since it's failing on "connect" and not "read", it's likely that there
> is a firewall between your client and the service you are trying to
> connect to which is dropping all packets instead of returning a
> "connection refused" response.
>
> So this isn't a problem with your code or with Tomcat. It's a problem
> between your client (which has the stack trace below) and the service
> that code is trying to call.
>
I am planning to use Wireshark to find out the root cause. Some weird
thing must be happening while making call through our client code.
Let's see.

> - -chris
>
> >
> > [1] java.net.ConnectException: Connection timed out: connect at
> > java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
> a:350)
> >
> >
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
> l.java:206)
> > at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> >
> >
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> > java.net.Socket.connect(Socket.java:589) at
> > sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
> > sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:472) at
> > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImp
> l.java:153)
> >
> >
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSo
> cket(SSLProtocolSocketFactory.java:82)
> > at
> > org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.create
> Socket(SSLProtocolSocketFactory.java:127)
> >
> >
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
> 7)
> > at
> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
> MethodDirector.java:387)
> >
> >
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
> dDirector.java:171)
> > at
> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
> :397)
> >
> >
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
> 23)
> >
> > [2] "http-apr-18100-exec-5" #327 daemon prio=5 os_prio=0
> > tid=0x2efbf800 nid=0x2508 runnable [0x2e55d000]
> > java.lang.Thread.State: RUNNABLE at
> > java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
> a:350)
> >
> >
> - - locked <0x00071a984780> (a java.net.TwoStacksPlainSocketImpl)
> > at
> > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI
> mpl.java:206)
> >
> >
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
> 8)
> > at java.net.PlainSocketImpl.connect(

OT: Connection timeout with HttpClient

[Suvendu Sekhar Mondal]

One of our legacy applications is using Apache Commons HttpClient 3.1.
POST call to one REST service is failing with
"java.net.ConnectException: Connection timed out: connect"
exception[1]. Timeout is occurring after one minute. To figure out
where thread is spending all the time, I took multiple thread dumps.
In all of them, thread is trying to create secure socket[2]. I am not
seeing any thread pool issue. Also the target REST service is working
properly with same payload while invoked from Postman. It is only
failing consistently when going through the HttpClient. I am trying to
figure out why it is taking so long in connecting to the socket. I am
looking for suggestion on how to attack this one.

Tomcat: 7.0.55
JRE: 1.8_92
-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

[1]
java.net.ConnectException: Connection timed out: connect
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:472)
at 
sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:153)
at 
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:82)
at 
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:127)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)

[2]
"http-apr-18100-exec-5" #327 daemon prio=5 os_prio=0
tid=0x2efbf800 nid=0x2508 runnable [0x2e55d000]
   java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x00071a984780> (a java.net.TwoStacksPlainSocketImpl)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:472)
at 
sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:153)
at 
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:82)
at 
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:127)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: error 0 issue

[Suvendu Sekhar Mondal]

On Wed, Jun 26, 2019, 11:02 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Kumar,
>
> On 6/26/19 13:27, Kumar R wrote:
> > Hi, Thanks for your reply...! As the application is available for
> > users, also cost is involved for new Linux VM.
>
> ??!
>
> > Is it possible with existing window VM 32 bit.
>
> It is not possible to safely run your application in the environment
> you have described.
>
> - -chris
>

Chris,
Agreed.

Kumar,
You are limited by the 32-bit OS architecture. There, a process can have
max 2^32 Bytes of virtual memory. Out of that, 2gb will be system address
space and rest of the user address space. There is a 3 gb switch by using
which you can increase user address space by 1gb. But again that can make
system unstable as you are cutting down system address space. We were in
this situation and moved to 64-bit architecture 7-8 years ago. If you just
want to upgrade, I believe you can use 32-bit version of Java and Tomcat.
But if you want more memory power, you have to move to 64-bit environment.

Thanks!

> On Wed, Jun 26, 2019, 10:49 PM John Larsen
> >  wrote:
> >
> >> Why windows - especially from 2003. If app is in java you'll get
> >> huge performance boost moving to linux.
> >>
> >> John Larsen
> >>
> >>
> >> On Wed, Jun 26, 2019 at 11:11 AM Kumar R 
> >> wrote:
> >>
> >>> Hi Team, Is it possible to go for higher version of JDK(64 bit)
> >>> and Tomcat(64bit)
> >> on
> >>> 32 bit window 2003 architecture. If so, kindly let me know the
> >>> possible difficulties. Thanks Rajib
> >>>
> >>> On Tue, Jun 25, 2019, 1:17 AM Kumar R 
> >>> wrote:
> >>>
>  Hi, Thanks for the help. Thanks & Regards Rajib
> 
>  On Tue, Jun 25, 2019, 1:14 AM Felix Schumacher <
>  felix.schumac...@internetallee.de> wrote:
> 
> >
> >
> > Am 24. Juni 2019 21:23:24 MESZ schrieb Kumar R
> >  >>> :
> >> Hi Team, I am facing server 0 issue while starting tomcat
> >> 5 service after increase the heap size from 1024 to
> >> 2048.
> >>
> >> Server:- Windows 32 Jre:- 1.5.0_15-h04, mixed mode
> >> sharing
> >
> > The 32 bit version of Java under windows can't use more
> > than about 1.5
> >>> GB
> > of ram.
> >
> > Note that the versions of Java and tomcat are way out of
> > date. Please
> >> do
> > yourself a favor and update them.
> >
> > Felix
> >
> >
> >> Tomcat:- 5.5 Error:- Jakarta log:- create JavaVM failed,
> >> failed initializing java. Event log:- The Apache Tomcat
> >> service terminated with services-specific error 0(0x0)
> >> Thanks Rajib
> >
> > --
> - ---
> >
> >
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail:
> > users-h...@tomcat.apache.org
> >
> >
> >>>
> >>
> >
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl0TrCIACgkQHPApP6U8
> pFg8ihAAuYrnii6Ho18Gg8s0Jz5UBQnpl6VwX+7o9BYil3NeBRZBL3MMCpriZBlS
> QBioRtpz/GpCo2Hu13pUQ7gI1I1DodjFxY69Uyr7rQ7Yi5k2+2yzn0hnJmQekser
> q4TPi2HLnYiZLcfk8i7rZjkV4W2dOYZGShecFM/osTzQdlsS8n8XHbpVt7f4nd3B
> a0F6T1JnMCsGIIq1mf2VvCOUYhS20j4A/FHjZ61TfdFhrfChz+ALoyDIL9tgK8AH
> zz2szpD3bMfz0jfEoYfCqvQvlqUDC8jZYsjvXXparHDLgBTQh//2X/T0SiMHcijH
> KoKSjZh4Zad56Xl45jXqMWETdHzy/VKArHiD5rB80D8MLvgspcOqVSr7pH5pHKq7
> JQELBxnzgibZLv7qP8CRr0Osc1m6X0qby82MJpkP6pBOeAR+Wu/kDDc3RO11gox8
> +DnLjRciDcVbVPCGXXjJPTh8zGNCcNOs6liyzkPlZmmm3IrtG8LdsRQs7frGTay3
> RnWnORU3W1xBPQ4M9/b9MPCnqVH/l+lW1CMAOUrsZwK6OdfhPohfsCd9xXipRbKc
> pUivjLGMANCeJgvgyUyzBvVlcuPlkNu4VPyAC6/yomBm+P8fHP4A/CYGqXTlQ5KT
> egn7ycbs1txQjjotaw54XQQIxkfzSdCkpYWaWTl0xMaDGT/0OhA=
> =OtHe
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Queries regarding tuning Apache Tomcat to serve high traffic

[Suvendu Sekhar Mondal]

On Wed, Jun 12, 2019, 7:18 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Vinu,
>
> On 6/12/19 04:53, Vinu Vibhu Sobhana wrote:
> > Hai Team,
> >
> > I have been requested by Project Manager to host a web application
> > in Tomcat 8.5 and tune the server to cater high traffic. So, I
> > have hosted the web application on an allotted VPS server with
> > specification (8 vCPU, 16GB RAM and 200 HD), with Tomcat 8.5.42
> > installed.  The web application has been successfully hosted also.
> >
> > Now, with respect to tuning part, I have to clear some points
> > before making it on-line
> >
> > 1. How can I tune this server to cater high traffic.
>
> Tomcat is tuned fairly well out of the box.
>
> > 2. What all are the parameters that I need to considered while
> > tuning
>
> There really aren't that many things you can change. Mostly, you can:
>
> 1. Add more threads to the thread pool
> 2. Allow more connections
> 3. Adjust timeouts to evict clients who aren't responding quickly enough
>
> > 3. Any open-source tools to generate high traffic and test the
> > currently configured server
>
> Apache JMeter is pretty much the standard, here. Remember that you
> want to make sure that your test is valid by generating load from
> multiple servers and not just one host which might limit itself.
>
> Our testing[1] indicates that you can fill the network before Tomcat
> begins to struggle with moving bytes around.
>
> Your application is far more likely to be a bottleneck than anything
> Tomcat related.


> > 4. Any references / biogs that I can refer regarding this topic 5.
> > Any suggestions from your end.
>
> It sounds like you have a single server running your application. I
> would recommend more than one for fault-tolerance and
> high-availability. Users are happier with a slower, available service
> than a fast one that isn't reachable. :)

> Note: A brief about the the web application. Its a small
> > recruitment portal web application that lets 1. Clients to register
> > their details 2. Allot a time-frame to have an on-line exam 3.
> > Performs on-line exams for the selected candidates (another
> > schedule) 4. Publish their results 5. Expected traffic rate 1000
> > hits/sec approx.
>
> That doesn't give us much information. What really matters is the
> amount of information going over the network. Tomcat mostly just
> pushes bytes over the network for you and dispatches everything to
> your application.
>
> So unless you are having a specific problem, there really is nothing
> to do with Tomcat to make it "faster". It's already pretty much as
> fast as it can be.
>

+1

There is no "silver bullet" to tune and scale one app. In my 10 years of
(little)work experience I have seen that mostly apps are the bottleneck,
not the web containers. How much one app can scale is hugely depends on how
much time it is taking to perform a single unit of work. If it is in few
millisecond range, probably you are in good shape. IMO, anything taking
more than a second or two in business logic processing, is a candidate for
tuning. Sometimes it's really tough to tune legacy app. In those cases I
follow "first parallelize, then optimize" rule. Create more JVM
instances(redundant app deployment), do parallel processing, reach
scalability target and at the very same time work to optimize the code.
This redundancy will make your app more fault tolerant and resilient. But
again after some level, you will hit some other bottleneck. So, tune your
app as much as you can and hopefully everything will start flying. :)


> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl0BApoACgkQHPApP6U8
> pFgq3xAApCqyUS3DUVwF2PNwhNStms7Y8gbW9A53PInqeu9c7vyM6JN2e8DexccG
> N9ZTZXv8EEYFXSJJasad3S7d57pVON/TgCK/+XNcFLMSajn/6vhNJF6j6WoqRLmo
> rbkSFylG6rKytn1vd8fgw7/o3yeahUbyDHVwn3hY3ZTHXYVO9tM2zgnHe65FB7pO
> M/N9a52NXeObvt9KrssMVHjH7cDBTh6gOAOKq7nVD8qi7S6nU4SLBJn7bhqNRwJc
> vTQfJgq1jEKe2c3O2ynbWckzXzRCqq/rmFWAU8srsdVbBxikzz/W2kIjsWxwQf5s
> sIbzh3ZB03XE0ko+Dnr16DmZMK7uWW+J2IAoQr+XOJnIkz6t7DuV3q/8q0+Q9gnf
> fXya+xK5PJsOoKxxPyqOVw/C6ZdGoPwJDDVlpJR8RrfdozL+zycRC8uIXoHOvr7H
> PoDDwnfzR410d5mXt0vBSXbndJZl/Bjv68bfUfHi5RvJr4GYJtgVqHl71cLD6aSy
> WGacv4rBe+B5LWexDso/Ajb3aE/DB7at+EpB4ZThKlla4Et0K5EVOZ71YAuSfbIO
> /IvSXpZcwa5b1YIZ0Fdrnv5QpBdY0SyuRB8ImhUcqQi46VoYdyao1UbjN72aYPX5
> YpaDnK+EDMLrxOBwdURJUMEUWJUWjrlSvVsZ4DmxqDUBWN4HFDU=
> =q2wh
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Massive Startup Time after Server Reboot

[Suvendu Sekhar Mondal]

Jerry,


On Tue, May 21, 2019, 10:55 PM Jerry Malcolm  Update I totally disabled Windows Real Time Virus protection.
> Apparently it was 'one' of the problems.  However...
>
> Original startup after reboot (with WIndows Real Time Virus scan) -- ~21
> minutes
>
> With Windows Real Time Virus scan totally disabled immediately after
> reboot -- ~3 minutes
>

Wow!

>
> Second TC startup -- ~35 seconds.
>
> 3 minutes beats 21 minutes, and I guess I can live with it.  But there's
> still something else slowing it down on the first startup after system
> reboot.   I'll see about getting some stack dumps.
>

May I know what is the startup type of Tomcat service? If it is
'automatic', can you please try with 'automatic delayed start'?

>
> Thanks for the help.
>
> Jerry
>
>
> On 5/21/2019 9:55 AM, Christopher Schultz wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > Jerry,
> >
> > On 5/20/19 14:05, Jerry Malcolm wrote:
> >> Andre,
> >>
> >> Your theory sounds like a real possibility.  It would explain why
> >> it only blows up the first time after a reboot.  I'll test the
> >> theory tonight during off-peak use hours for the web sites I host.
> > If André's theory is correct, you will probably find that 2-3 months
> > ago the version of your virus scanner was upgraded, and all of the
> > whitelisted applications (like Tomcat) were lost.
> >
> > If that's the case, you can probably re-whitelist Tomcat and get your
> > performance back. At least until the next upgrade.
> >
> > - -chris
> >
> >> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:
> >>> On 20.05.2019 00:09, Jerry Malcolm wrote:
>  Just an FYI this server has been in production a little
>  over a year.  This TC load problem only started 2-3 months ago.
>  So 'something' changed to cause this problem to manifest
>  itself. It's been too long to try to correlate a specific
>  server change to the start of this problem. But I just wanted
>  to point out that this problem has NOT been around since day 1
>  on this server.
> >>> Just in case, since this is a Windows server .. 2-3 months ago,
> >>> an update of a virus scanner ? (of the intrusive kind which
> >>> pre-checks each file that wants to be opened) Checking may be
> >>> easy or not, depending on your access : disable the virus scanner
> >>> just the time to start tomcat and check.
> >>>
>  Jerry
> 
>  On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
> > Rainer,
> >
> > No change with the urandom parm.  I am attaching a portion of
> > the Catalina log.  The first half shows between 8 and 15
> > seconds to deploy each app for a single virtual host (there
> > are no war files, the app is already exploded in the appbase
> > dir).  I have quite a few virtual hosts with several apps
> > each.  That initial server start took 21+ minutes. I then
> > restarted the TC service and got the ~.5 sec start per webapp
> > shown in the 2nd half of the log below.
> >
> > Can you refresh me on how to capture the stack dumps you
> > suggested? It's been a while
> >
> > Thx.
> >
> > Jerry
> >
> > First start of Tomcat after server reboot
> > ---
> > - --
> >
> > [C:\domains\.com\webapps\JSPWiki.war] has finished in
> > [8,579] ms tory [C:\domains\.com\webapps\cis]
> > irectory [C:\domains\.com\webapps\cis] has finished
> > in [11,486] ms tory
> > [C:\domains\.com\webapps\gallery] irectory
> > [C:\domains\.com\webapps\gallery] has finished in
> > [9,204] ms tory [C:\domains\.com\webapps\gl]
> > irectory [C:\domains\.com\webapps\gl] has finished
> > in [8,469] ms tory
> > [C:\domains\.com\webapps\idmanager] irectory
> > [C:\domains\.com\webapps\idmanager] has finished in
> > [8,689] ms tory
> > [C:\domains\.com\webapps\itemtrack] irectory
> > [C:\domains\.com\webapps\itemtrack] has finished in
> > [6,907] ms tory
> > [C:\domains\.com\webapps\malcolment] irectory
> > [C:\domains\.com\webapps\malcolment] has finished
> > in [8,469] ms tory
> > [C:\domains\.com\webapps\notebook] irectory
> > [C:\domains\.com\webapps\notebook] has finished in
> > [10,189] ms tory [C:\domains\.com\webapps\order]
> > irectory [C:\domains\.com\webapps\order] has
> > finished in [8,501] ms tory
> > [C:\domains\.com\webapps\payment] irectory
> > [C:\domains\.com\webapps\payment] has finished in
> > [14,209] ms tory
> > [C:\domains\.com\webapps\projectmanager] irectory
> > [C:\domains\.com\webapps\projectmanager] has
> > finished in [9,018] ms [C:\Tomcat
> > 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat
> > 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in
> > [62] ms
> >
> > Restart of Tomcat
> > ---
> > - --
> 

Re: Massive Startup Time after Server Reboot

[Suvendu Sekhar Mondal]

On Mon, May 20, 2019, 5:46 PM Louis Zipes  Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while
>
> >In Task Manager find the Tomcat process, right click on it and there is
> an option to create a dump file.
>

Well, that will create Windows memory dump.

>
> >Also, if you have access to jconsole or other JMX tool you can connect
> to it and see what is going on.
>

Agreed. Alternatively you can use jstack to generate thread dumps. Exact
command will be like: jstack -l PID_of_JVM

I'll suggest to take thread dumps 1 sec apart.

But still I am surprised to see that it happens only during entire server
reboot and from your log, subsequent TC restart is not taking long time. Is
that a right statement or I am entirely misreading it? If that is true, you
can try 'atomatic delayed start' option.

>
> -Original Message-
> From: André Warnier (tomcat) [mailto:a...@ice-sa.com]
> Sent: Monday, May 20, 2019 4:44 AM
> To: users@tomcat.apache.org
> Subject: Re: Massive Startup Time after Server Reboot
>
> - - - external message, proceed with caution - - -
>
>
> On 20.05.2019 00:09, Jerry Malcolm wrote:
> > Just an FYI this server has been in production a little over a
> year.  This TC load
> > problem only started 2-3 months ago.  So 'something' changed to cause
> this problem to
> > manifest itself. It's been too long to try to correlate a specific
> server change to the
> > start of this problem. But I just wanted to point out that this problem
> has NOT been
> > around since day 1 on this server.
>
> Just in case, since this is a Windows server ..
> 2-3 months ago, an update of a virus scanner ?
> (of the intrusive kind which pre-checks each file that wants to be opened)
> Checking may be easy or not, depending on your access : disable the virus
> scanner just the
> time to start tomcat and check.
>
> >
> > Jerry
> >
> > On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
> >> Rainer,
> >>
> >> No change with the urandom parm.  I am attaching a portion of the
> Catalina log.  The
> >> first half shows between 8 and 15 seconds to deploy each app for a
> single virtual host
> >> (there are no war files, the app is already exploded in the appbase
> dir).  I have quite
> >> a few virtual hosts with several apps each.  That initial server start
> took 21+ minutes.
> >>   I then restarted the TC service and got the ~.5 sec start per webapp
> shown in the 2nd
> >> half of the log below.
> >>
> >> Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while
> >>
> >> Thx.
> >>
> >> Jerry
> >>
> >>  First start of Tomcat after server reboot
> >> -
> >>  [C:\domains\.com\webapps\JSPWiki.war] has finished in
> [8,579] ms
> >> tory [C:\domains\.com\webapps\cis]
> >> irectory [C:\domains\.com\webapps\cis] has finished in
> [11,486] ms
> >> tory [C:\domains\.com\webapps\gallery]
> >> irectory [C:\domains\.com\webapps\gallery] has finished in
> [9,204] ms
> >> tory [C:\domains\.com\webapps\gl]
> >> irectory [C:\domains\.com\webapps\gl] has finished in [8,469]
> ms
> >> tory [C:\domains\.com\webapps\idmanager]
> >> irectory [C:\domains\.com\webapps\idmanager] has finished in
> [8,689] ms
> >> tory [C:\domains\.com\webapps\itemtrack]
> >> irectory [C:\domains\.com\webapps\itemtrack] has finished in
> [6,907] ms
> >> tory [C:\domains\.com\webapps\malcolment]
> >> irectory [C:\domains\.com\webapps\malcolment] has finished in
> [8,469] ms
> >> tory [C:\domains\.com\webapps\notebook]
> >> irectory [C:\domains\.com\webapps\notebook] has finished in
> [10,189] ms
> >> tory [C:\domains\.com\webapps\order]
> >> irectory [C:\domains\.com\webapps\order] has finished in
> [8,501] ms
> >> tory [C:\domains\.com\webapps\payment]
> >> irectory [C:\domains\.com\webapps\payment] has finished in
> [14,209] ms
> >> tory [C:\domains\.com\webapps\projectmanager]
> >> irectory [C:\domains\.com\webapps\projectmanager] has
> finished in [9,018] ms
> >>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
> >> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has
> finished in [62] ms
> >>
> >>  Restart of Tomcat
> -
> >> tory [C:\domains\.com\webapps\cis]
> >> irectory [C:\domains\.com\webapps\cis] has finished in [594]
> ms
> >> tory [C:\domains\.com\webapps\gallery]
> >> irectory [C:\domains\.com\webapps\gallery] has finished in
> [547] ms
> >> tory [C:\domains\.com\webapps\gl]
> >> irectory [C:\domains\.com\webapps\gl] has finished in [562] ms
> >> tory [C:\domains\.com\webapps\idmanager]
> >> irectory [C:\domains\.com\webapps\idmanager] has finished in
> [578] ms
> >> tory [C:\domains\.com\webapps\itemtrack]
> >> irectory [C:\domains\.com\webapps\itemtrack] has finished in
> [547] ms
> >> tory [C:\domains\.com\webapps\malcolment]
> >> irectory [C:\domains\.com\webapps\malcolment] has finished in
> [579] ms
> >> tory 

Re: Issues upgrading to tomcat 9.0.17

[Suvendu Sekhar Mondal]

On Wed, Apr 3, 2019, 10:07 PM Kim, Chang H (JMD)
 Yes, that's correct.  The same browser, hitting the same url since both
> tomcat 8 and 9 are installed on the same server.  Tomcat 8 works, but
> tomcat 9... blank.
>

As per your access log sample, Tomcat is not getting the request. Can you
please use something like Fiddler to confirm that your request has reached
Tomcat? And please pay attention to the HTTP status of the request, whether
you are getting 200 or it simply hangs.


> Thanks,
>
> Kyle Kim
> JMD
>
> Confidentiality Notice:  This e-mail, including all attachments, is
> intended only for the sole use of the intended recipient(s) and may contain
> privileged and/or confidential information.  If you are not the intended
> recipient(s) of this e-mail, any dissemination, distribution or copying of
> this e-mail, and any attachment(s) thereto, is strictly prohibited and may
> violate Federal Law.  If you have received this e-mail in error, please
> immediately notify the sender by e-mail or telephone and permanently delete
> all copies of this e-mail and any attachment(s).
>
>
> -Original Message-
> From: André Warnier (tomcat) 
> Sent: Wednesday, April 3, 2019 12:14 PM
> To: users@tomcat.apache.org
> Subject: Re: Issues upgrading to tomcat 9.0.17
>
> On 03.04.2019 17:57, Kim, Chang H (JMD) wrote:
> > Yes, I see "GET" when I use my old tomcat 8.0.35.  However, my newly
> installed 9.0.17, nothing...
>
> Are you using the same browser/client in both cases ?
> And are the connections to the old and new Tomcats the same also ? (I
> mean, are they in the same place, and are there the same in-between
> "pieces" - such as proxies, firewalls,..)
>
>
> >
> > Thanks,
> >
> > Kyle Kim
> > JMD
> >
> > Confidentiality Notice:  This e-mail, including all attachments, is
> intended only for the sole use of the intended recipient(s) and may contain
> privileged and/or confidential information.  If you are not the intended
> recipient(s) of this e-mail, any dissemination, distribution or copying of
> this e-mail, and any attachment(s) thereto, is strictly prohibited and may
> violate Federal Law.  If you have received this e-mail in error, please
> immediately notify the sender by e-mail or telephone and permanently delete
> all copies of this e-mail and any attachment(s).
> >
> >
> > -Original Message-
> > From: André Warnier (tomcat) 
> > Sent: Wednesday, April 3, 2019 11:53 AM
> > To: users@tomcat.apache.org
> > Subject: Re: Issues upgrading to tomcat 9.0.17
> >
> > On 03.04.2019 17:45, Kim, Chang H (JMD) wrote:
> >> I had to remove the ip specific data, but this is what I am seeing in
> localhost_access_log.*.txt when I see "blank screen".
> >>
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:14 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:14 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:18 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:19 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:23 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:24 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:28 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:29 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:33 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:34 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:38 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:39 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:43 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:44 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:48 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:49 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:53 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:54 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:58 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:41:59 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:03 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:04 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:08 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:09 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:13 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:14 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:18 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:19 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:23 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:24 -0400] "HEAD / HTTP/1.0" 200 -
> >> XX.XX.XX.XX - - [03/Apr/2019:11:42:28 -0400] "HEAD / HTTP/1.0" 200 -
> 

Re: Thread pools

[Suvendu Sekhar Mondal]

Hi Rajendra,

On Fri, Mar 29, 2019 at 2:44 AM Rajendra Popuri
 wrote:
>
> Hi,
>
> Is any parameter to recycle Tomcat connector thread pools like Datasource
> pools? Or we can only define maxThreads value to connector element. Please
> advise.
>

Can you please elaborate on "recycle Tomcat connector thread pools"
part? Exactly what you wanted to do, to purge idle threads or
something else.

There are different Connectors properties. Please see[1].

[1] https://tomcat.apache.org/tomcat-7.0-doc/config/http.html

Thanks!
Suvendu

> Thanks,
> Rajendra
> --
> Thanks Rajendra

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Usage

[Suvendu Sekhar Mondal]

Lucas,

On Thu, Mar 21, 2019, 11:08 PM Igal Sapir  Lucas,
>
> On 3/21/2019 6:57 AM, Lucas Bovetto wrote:
> > Hello,
> >
> > I whould like to get a support to a problem that has been happening.
> > I have scenario that my tomcat stay in 100% cpu usage even though no
> > one access, and when it happening nobody can accesses tomcat.
> >
> > Configuration:
> > O.S: Windows Server 2008
> > Tomcat: 8.5
> > Japa Options:
> > -XX:+UseParNewGC
> > -XX:+OptimizeStringConcat
> > -XX:NewSize=1G
> > -XX:+CMSParallelRemarkEnabled
> > -XX:+UnlockDiagnosticVMOptions
> > -XX:-DisableExplicitGC
> > -XX:SoftRefLRUPolicyMSPerMB=1
> > -XX:SurvivorRatio=20
> > -Djava.awt.headless=true
> > -Djava.net.preferIPv4Stack=true
> > -XX:MaxPermSize=512m
> > -XX:+UseTLAB
> > -XX:MaxTenuringThreshold=0
>
> We need more information before we can help:
>
> What version of Java are you using?
>
> Can you issue a Thread Dump?
>
> How much RAM does the Tomcat process take when the issue happens?  How
> much physical RAM does the machine have?
>
> Did you set all of these -XX options after research and conclusion that
> they are optimal for your setup or did you just copy and paste them from
> a random post?
>
> For example, if you are running Java 8 then the MaxPermSize is ignored
> because PermGen was replaced with MetaSpace.  You also set explicitly
> the NewSize and the SurvivorRatio and at first glance, without knowing
> anything about your setup, the values don't seem right to me.
>
>
> Igal
>
>
> >
> > Att,
> >
> >
> > --
> >
> > *Lucas Henrique Bovetto*
> >
> > *Gerente de Desenvolvimento*
> >
> > _
> >
> > *_Imagem inline 2_*
> >
> > *_www.mkdata.com.br _*
> >
> > *Av. Campos Sales, 420 – Jd. Girassol***
> >
> > *Americana – 13465-590 – SP***
> >
> > *(19) 3407-7447*
> >
> >
> > Imagem inline 3
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org


There is high chance that some "rouge" thread(s) is causing this problem.
During problem period, I will suggest to use ProcessExplorer to identify
thread ID which are taking most of the CPU. You can get that information
from Threads tab of ProcessExplorer. At the same time please take at least
3 thread dumps 5 seconds apart. After that convert the Decimal value of
thread ID to Hexadecimal. Search that value in the captured thread dumps.
You should see them matching with native thread ID. Check if the thread is
moving at all or not. Repeat this process for all threads which are taking
most of the CPU. By that way you will get to know what actual problem is.
You can share the details with this community.

Happy hunting! :)


>

Re: Application hanging on Tomcat 7.0.54

[Suvendu Sekhar Mondal]

On Thu, Sep 27, 2018, 6:20 PM Louis Zipes  wrote:

> Chris,
> I looked through the log some more and I see all of the types of Thread
> Statuses.  Blocked, Runnable, Waiting, etc..  Any in particular that I
> should concentrate on?
>

Louis, can you please take multiple thread dumps(at least 3) with 5sec
interval? You need to look for threads which are not moving at all or
moving very slowly. They could be in any state. In worst case scenario you
might see some blocking or deadlock. That will give you lead on what's
going on inside the container. You can use IBM's thread dump analyzer for
that.

Ex.
> "http-bio-7005-exec-128" daemon prio=6 tid=0x26466800 nid=0x40e4
> waiting for monitor entry [0x432ae000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.demantra.applicationServer.servlets.notifications.UserNotificationHelper.execute(UserNotificationHelper.java:117)
> - waiting to lock <0x00054d652c08> (a
> com.demantra.applicationServer.metaDataObjects.user.UserList)
>
> ODBC on the actual Window machine hosting Tomcat is Oracle in
> OraClient11g_home1  (we have a 12c Oracle database) with a pool timeout set
> to 60.  Should I be looking to turn on some tracing on the driver?
>
> Thanks, Louis
>
>
>
> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Wednesday, September 26, 2018 7:30 PM
> To: users@tomcat.apache.org
> Subject: Re: Application hanging on Tomcat 7.0.54
>
> - - - external message, proceed with caution - - -
>
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Louis,
>
> On 9/26/18 15:56, Louis Zipes wrote:
> > Problem just re-occurred and so I was able to at least get a JSTACK
> > (I assume it was Tomcat since it was the Java using the most memory
> > on the machine).  Here is the reoccurring message.  I get more hits
> > on but haven't dug through all of the Google hits yet (due to
> > multi-tasking) so apologies up front if there is a simple answer to
> > this.
> >
> > "Event_Manager_1413" daemon prio=6 tid=0x24856000
> > nid=0x40c4 waiting on condition [0x42dae000]
> > java.lang.Thread.State: TIMED_WAITING (parking) at
> > sun.misc.Unsafe.park(Native Method) - parking to wait for
> > <0x0005ab45f7b8> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >
> >
> at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
> > at
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.
> awaitNanos(Unknown
> > Source) at java.util.concurrent.LinkedBlockingQueue.poll(Unknown
> > Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown
> > Source) at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source) at java.lang.Thread.run(Unknown Source)
>
> This thread is waiting for a task, and is essentially idle. You will
> have many of these on a non-busy system.
>
> What are the other threads doing?
>
> > Locked ownable synchronizers: - None
> >
> >> Any comments/suggestions are appreciated!
> >
> > Your most likely problem is database connection pool
> > mismanagement: connections aren't properly released and the pool
> > empties. All threads are left waiting on available database
> > connections which will never be replenished.
> >
> > I'm using the ojdbc6.jar if that is what you are referring to or is
> > there a better setting somewhere.
>
> ODBC? What is your database?
>
> - -chris
>
>
> > -Original Message- From: Christopher Schultz
> > [mailto:ch...@christopherschultz.net] Sent: Wednesday, September
> > 26, 2018 3:46 PM To: users@tomcat.apache.org Subject: Re:
> > Application hanging on Tomcat 7.0.54
> >
> > - - - external message, proceed with caution - - -
> >
> >
> > Louis,
> >
> > On 9/26/18 14:42, Louis Zipes wrote:
> >> Hi all, Tomcat 7.0.54 running on Windows 2012
> >
> >> We are running a third party application on Tomcat and today we
> >> have intermittently run in issues where the application stops
> >> working.  The big changes in our system is that we have added
> >> more end users and we are at year end so of course everyone is
> >> hitting the system hard. Even if we force a log out of all users
> >> and stop all background jobs then the application doesn't
> >> recover.
> >
> >> We see no active sessions on the database (our application is
> >> connecting to an Oracle database) and I see no clear error
> >> messages in either our third party application logs or the Tomcat
> >> logs (ex. OutofMemory).  When we go to the Windows Task Manager
> >> we did not see the machine's Memory max'd out but admittedly I
> >> didn't look at the Java session to see if was reaching its Heap
> >> Max.  The only thing that we noticed was that TCP connections
> >> went down right after the restart.  I did open up Jconsole under
> >> Java and I did force a garbage collection but that didn't seem to
> >> help.
> >
> >> We 

Is this a classloader leak?

[Suvendu Sekhar Mondal]

Hello Everyone,

Recently I am investigating a Metaspace leak issue. I dumped Metaspace
content over time using jcmd. While comparing them I can see that
there are 15,000 classes which are taking 120MB in that area! For all
of them InstCount(number of object instances of the Java class) is
zero. That tells me there is *no object* for those 15K classes in
heap. I can see that too in heap. No objects found for those classes.

Then why are they still live in the Metaspace and not getting cleared
by Full GC? Definitely they have some reference(s) from somewhere. All
I can see that they were loaded by 'class loader 0x21005760a
'org/apache/catalina/loader/WebappClassLoader'. Is this another form
of classloader leak or something else? Any idea?

Environment:
Windows 2012 server, JDK 1.8_92, Tomcat 7.0.55

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat service gets stuck

[Suvendu Sekhar Mondal]

On Thu, Aug 23, 2018, 2:06 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Bab,
>
> On 8/22/18 9:49 AM, Bab Alemzadeh wrote:
> > I am using Windows Server 2008/2012, datacenter edition. We have
> > installed: KB4338815.
> >
> > So the task gets stuck, it is probably not consuming any CPU. But
> > the memory usage gets stuck. I enclosed the screenshot, this
> > screenshot is AFTER the tomcat service was stopped.
>
> Screenshots are removed messages posting to the list. Please find
> another way to express the status of your system.
>
> > The memory usage is still at ,9 MB even though the Tomcat
> > service is completely stopped!
>

Bab,
This is an interesting issue. Couple of questions:

Are you running Tomcats as a published app in Citrix?

After restart, are you seeing another process started on same xenapp
server?Just wondering how is that possible as port is static.

Can you please give more details about the 'stuck' Tomcat process? 3 Thread
dumps in 5sec intervals would be great. From your description it seems that
process entering in hang state. Please use -F switch of jstack to dump
threads.

Are you still able to connect to the hang Tomcat using JMX? What about the
2nd instance, is that accessible via JMX?

Please let us know.

1GiB isn't much for a non-trivial web application.
>
> But please do post a follow-up with an explanation.
>
> - -chris
>
> > On Wed, Aug 22, 2018 at 6:32 AM, Christopher Schultz
> >  > > wrote:
> >
> > Bab,
> >
> > On 8/22/18 9:15 AM, Bab Alemzadeh wrote:
> >> I have a problem with tomcat 8.5.20 and 8.5.32. When I stop the
> >> service the PROCESS gets stuck on a specific memory usage.
> >
> > Can you explain this a little more? What does "stuck on a specific
> > memory usage" mean?
> >
> >> I cannot even end the task, I have to restart the server.
> >
> > Wow.
> >
> >> If I just restart the service, it will create an additional
> >> process. The other process will remain stuck at the exact same
> >> memory usage.
> >
> > More information, please.
> >
> >> I am using Tomcat on a Citrix environment.
> >
> > Citrix is unlikely to be a contributing factor, but thank you for
> > that information.
> >
> > I'm assuming this is a Windows environment (Citrix + "service")?
> > Can you please give us the version details?
> >
> > Specifically, have you installed KB4338815 or KB4338818?
> >
> > This might be your problem:
> > https://stackoverflow.com/questions/51666952/address-bind-exception-in
> - -t
> >
> >
>  t>
> > omcat
> >
> > -chris
> >
> > -
> >
> >
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >  For additional
> > commands, e-mail: users-h...@tomcat.apache.org
> > 
> >
> >
> >
> >
> > -- With kind regards/Met vriendelijke groeten,
> >
> > Bab Alemzadeh System Administrator
> >
> > EyeQuestion Software (Logic8 B.V.) The Netherlands Tel.:
> > 0031628735930 EQ   EQT
> > 
> >
> > Logic8 B.V. - Aamsestraat 90D - PO Box 206 - NL-6660 AE - Elst(Gld)
> > - The Netherlands The information contained in this communication
> > is confidential and may be legally privileged. It is intended
> > solely for the use of the individual or entity to whom it is
> > addressed and others authorized to receive it. If you are not the
> > intended recipient you are hereby (a): notified that any
> > disclosure, copying, distribution or taking any action with respect
> > to the content of this information is strictly prohibited and may
> > be unlawful, and (b): kindly requested to inform the sender
> > immediately and destroy any copies. Logic8 does neither accept any
> > responsibility and/or liability for the improper and incomplete
> > transmission of the information contained in this communication nor
> > for any delay in its receipt.
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlt9yToACgkQHPApP6U8
> pFiAVQ/7BhicmzVU6rjlOcAD+iDEXAcp+fHMz+Dlm4TFI9b3GvibylOX7P6yWWMA
> pdk/QSe7GqKh77PyAroHH5TLQbsuzorObdnD23mNK2rWm+ga1ttX8XqBpCW9bSZH
> Sr7tldC4pBL43Oh0T0uxa0iYONeXmJWf6mn0UYpVwKz75V2l7yLbuVviKpFL+bK0
> jaFzfpSEYzxuJMiIr6MC/Fywto1frPETz5+eU67nMUyujMMXQXUn2PHqU5pimBSM
> /JnV8PuPFl6fdPlkLBCC73ylKlz1y7/9Ze7nmRRDETaiCP6MOV6iF3SLaT0WSjXX
> 2UYUV9z2YfOdG+kGQ7kLCBDmwZuED1vi2ngrPAx7RliyK6JFqs1MBXHUJQ33Ch+7
> rnN1aURWB+SbEXsByWeDbA9m4joJUrp0ESnkjCXezOOh2EMY/BYgWkoEvKgB80F9
> oHzsY+vGmIobLEwgrRzwmVW/tktN2y7wfoANbepxPS8hTOyPI3u1aWGTsk66swPo
> WhqVqt3E1rGXy8Vjl/aODamv6wcMK78tzdV2tFFDQCaKhjw8jddcAntRWt+/hgAQ
> 

Re: Windows Service Weirdness

[Suvendu Sekhar Mondal]

On Sat, Aug 11, 2018, 12:33 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> All,
>
> Did something change with Windows recently?
>
> On this list and on SO there seem to be lots of questions about
> "Tomcat not releasing ports".
>
> Examples:
> https://stackoverflow.com/questions/51764300/tomcat-8-does-not-release-p
> ort-80-at-restart-on-windows
> 
> https://stackoverflow.com/questions/51791717/tomcat-8-will-not-startup-o
> n-free-port
> 
> https://stackoverflow.com/questions/51773720/windows-keeping-port-locked
> - -after-shutting-down-tomcat
>
> The symptoms all seem to be the same: Tomcat starts, stops, but then
> won't restart using the same selection of ports. "netstat" shows that
> the ports are available, but launching Tomcat encounters a failure
> during port-binding.
>
> I don't have an environment in which to test this (I'm allergic to
> Windows), but I'd be interested to hear if anyone has seen anything
> like this, or if it's just very common to misconfigure Tomcat in some
> specific way (e.g. declaring two `` elements with the same
> port number).
>
> - -chris
>

Yes, Chris. Recently Microsoft released July security patches for Windows
OS. Now some of those patches are "buggy". After applying them, we started
seeing lots of TCP port conflict issues. In our case, Apache httpd as well
as Tomcats were affected. Symptoms matches what you mentioned in the mail.
netstat will show ports are free but TC will not be able to bind to it. So,
we decided to uninstall those patches and after that everything came back
to normal. Here is some details:
https://www.computerworld.com/article/3290465/microsoft-windows/stung-by-a-festering-pile-of-bugs-on-patch-tuesday-ms-releases-27-more-patches.html

To fix this issues, MS has released another set of patches. :)

Can you please ask the affected users to uninstall July security patches
and try again?

Thanks!
Suvendu

-BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAltt4ZIACgkQHPApP6U8
> pFh8+w//Y6UoJvFiLYLkwvJosPDLhn3TLgY+fGdpvo3KZKxTvBX2neLHRqLiJimt
> PVcc+5HipDut3Eu+bJLDPFKQSHN0WbG/IxkcUGyui2K4sUFM1WWxi1EWeHNqSNfj
> kpbIr/DnlhJSM0wVM5EH0lV0HO90UJ6tTNHPvQx03T+stuILPasmFXqoSNFCP3rC
> s9ZVk7LVltOPxPJ3tkqKLLOl2b0iaXltHXofC4GtEC/1BUEEYspxkBaCM9kOOsgl
> cWN9fGUAXgnQv+7ISStSl9BwvHk0Vf9WSaSYtPNzoQxa34CWmOls2gdUONAlIwyi
> 0ozLCHWPImR1d44DhDor6j0Uw7sQ5yFuoNs18ijcfzqdHYHqYqQ3RPGabowm8JQd
> pNvvaSGeHVEvdG2YHa0hpwLbDhchIL9/VcaQYiGHZk7HQVmbvCQ5gGGP5rKo6yPM
> E2uHEBDXdsiHuSd2rOpehc5Ysv1m0Qv6fZN4KFWMvREGb5IGtwvTzmcZr3iXw4W8
> 4Dx+TPGufKybW6W2F24Ko4xmxLYkptEyBhAd5v/XYeRHNfin9dYTAxkZnH57vMYc
> b+lWCqQlzKWqc8bhXtJvSbG8L6Mk6WtOLvRiOyz96JywES6Ald8QaOXE0siPEmws
> CQhgetCTrbSJlml7t9ahbEsOdt2LaczRmgXpdKrB6b/QAg448TM=
> =/OiR
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Question about setting CATALINA_OPTS when starting Tomcat using a Windows Service in Tomcat 7.0.54

[Suvendu Sekhar Mondal]

On Thu, Aug 9, 2018 at 2:39 PM Mark Thomas  wrote:
>
> On 09/08/18 01:06, Daniel Savard wrote:
>
> > Louis,
> >
> > I believe you need to understand a bit more how things are working with
> > Java and the JVM.
>
> Actually Daniel, it is you who needs to understand things better.
>
>
> Louis,
>
> Clearly, when Tomcat is started a new JVM instance is created and it
> listens on the configured JMX ports.
>
> The problem is that when Tomcat is stopped another JVM instance is
> created (to send the stop message to the first) and that uses the same
> configuration. Hence it tries to open the same JMX port and fails
> because it is already bound.
>
> If you were running from the command line, the fix would be easy. Put
> the JMX options in CATALINA_OPTS and they'd only be used on start but
> not stop. (JAVA_OPTS are used on both start and stop).
>
> There is also a simple fix if running as a Windows Service. The Windows
> Service wrapper is simply a renamed version Apache Common Daemon. When
> running a Java program as a Windows service there are three ways it can
> be integrated.
>
> 1. jvm. The Windows service wrapper starts and embedded JVM using the
> provided parameters and then calls the start method on the appropriate
> class. To stop, it calls the stop method on the appropriate class in the
> embedded jvm.
>
> 2. Java. The Windows service wrapper starts a separate Java process with
> the provided parameters. On stop, a second Java process is started using
> the same parameters which is expected to communicate with the first
> process and stop it.
>
> 3. exe. Same as 2 but any executable can be used rather than java.exe.
>
> You see the error you are see because you are using Java mode. Switch to
> jvm mode and all should be well.
>
> Finally 7.0.54 is very old. I strongly recommend an upgrade at least to
> the latest 7.0.x release is not 8.5.x/9.0.x
>
> Mark
>
One question Mark, if I use
org.apache.catalina.mbeans.JmxRemoteLifecycleListener, would that work
as JVM mode or Java mode?

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.8 with JRE 10.0.2 x64 Windows

[Suvendu Sekhar Mondal]

Shailendra,

On Fri, Jul 20, 2018 at 12:35 PM Shailendra Kumar Verma
 wrote:
>
> Hello,
>
> I recently installed Tomcat 9.0.8 on Windows 2016 server with JRE 10.0.2 x64. 
> After installation, it is taking 95% of CPU and 4 GB of RAM without any calls 
> running to the box. Why?
>
> System has 32 GB of RAM and 6 CPU cores.
>
> Why is Tomcat is taking 95% of CPU at idle condition, that is main concern?
>
> Thanks,
> Shailendra
>

That sounds abnormal. I will suggest you to do following when this
problem is occurring:
1. Run ProcessExplorer
2. Open Tomcat process which is eating up CPU
3. Go to the Thread tab
4. Sort by CPU column
5. Note down thread IDs which are consuming most of the CPU. List top 5 of them.
6. Take thread dump at the same time. You need to be fast here unless
thread state might change and you'll not get the correct picture.
7. Convert those thread ID list to the equivalent Hexadecimal value.
8. Search for those Hex values in the collected thread dump. Hex
values will match with native ID of thread dumps.
9. Post all those stack traces with CPU usage details over here.

Thanks!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat HTTP Sessions exceeded

[Suvendu Sekhar Mondal]

Sorry, there was a typo on my last email.

On Wed, Jul 11, 2018, 3:08 AM Gopi Palla  wrote:

> Hi,
>
> We have two dc, in one of the dc tomcat http sessions are exceeded. we are
> using introscope to monitor tomcat application server and getting the
> alerts, we kept the threshold value as 850 in introscope
>

Do you mean in one of  the Tomcats you are seeing 850+ HTTP sessions and
they never get cleaned up? What is the problem your app is facing?


> Tomcat version is 8.0.39
>
> Do we need to do any configuration changes in server.xml/web.xml ?
>
> Can someone please guide what to do in this case ?
>
> Appreciate your help.
>
>
>
>
> Regards
> Gopi
>

Re: Tomcat HTTP Sessions exceeded

[Suvendu Sekhar Mondal]

Gopi,

On Wed, Jul 11, 2018, 3:08 AM Gopi Palla  wrote:

> Hi,
>
> We have two dc, in one of the dc tomcat http sessions are exceeded. we are
> using introscope to monitor tomcat application server and getting the
> alerts, we kept the threshold value as 850 in introscope
>

Do you mean in one of  the Tomcats 850+have HTTP sessions are alive and
they  never get cleaned up? What is the problem your app is facing?

>
> Tomcat version is 8.0.39
>
> Do we need to do any configuration changes in server.xml/web.xml ?
>
> Can someone please guide what to do in this case ?
>
> Appreciate your help.
>
>
>
>
> Regards
> Gopi
>

Re: Apache Tomcat 9.0.8 install check

[Suvendu Sekhar Mondal]

On Mon, Jun 18, 2018 at 6:23 PM, Shailendra Kumar Verma
 wrote:
> Where? Can you be more specific which key are you talking about?

Under same registry entry. Here is the full path for installed Tomcat
8 on my laptop:
HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software
Foundation\Tomcat\8.0\Tomcat8_1\InstallPath

There is another key called version - if you want to verify the version again:
HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software
Foundation\Tomcat\8.0\Tomcat8_1\Version



> Thanks,
> Shailendra
>
> -Original Message-
> From: Suvendu Sekhar Mondal 
> Sent: Monday, June 18, 2018 6:21 PM
> To: Tomcat Users List 
> Subject: Re: Apache Tomcat 9.0.8 install check
>
> ***CAUTION: This email originated from outside of the organization. Do not 
> open attachments or click links unless you recognize sender and know content 
> is safe.*** Forward suspicious email to suspici...@convergys.com 
> ___
>
> Shailendra,
>
> On Mon, Jun 18, 2018 at 5:45 PM, Shailendra Kumar Verma 
>  wrote:
>> Hello,
>>
>> I am trying to find out through registry checking whether or not Apache 
>> Tomcat 9.0.8 is already installed or not. If the below registry is not 
>> there, then my program installs Apache Tomcat 9.0.8 installer otherwise it 
>> moves on to other installation and completes. It's kind of prerequisite 
>> check program.
>>
>> HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software
>> Foundation\Tomcat\9.0\Tomcat9
>>
>> However, everytime program just proceeds to install Tomcat despite it is 
>> already installed? Am I checking wrong registry?
>>
>> Thanks,
>> Shailendra
>>
>
> That is the correct registry path for installed TC. Additionally you can also 
> try to read value of InstallPath.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache Tomcat 9.0.8 install check

[Suvendu Sekhar Mondal]

Shailendra,

On Mon, Jun 18, 2018 at 5:45 PM, Shailendra Kumar Verma
 wrote:
> Hello,
>
> I am trying to find out through registry checking whether or not Apache 
> Tomcat 9.0.8 is already installed or not. If the below registry is not there, 
> then my program installs Apache Tomcat 9.0.8 installer otherwise it moves on 
> to other installation and completes. It's kind of prerequisite check program.
>
> HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software Foundation\Tomcat\9.0\Tomcat9
>
> However, everytime program just proceeds to install Tomcat despite it is 
> already installed? Am I checking wrong registry?
>
> Thanks,
> Shailendra
>

That is the correct registry path for installed TC. Additionally you
can also try to read value of InstallPath.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: application goes down after restart the tomcat server 8.29 or restart the OS

[Suvendu Sekhar Mondal]

Loai,

On Thu, Jun 7, 2018, 2:03 AM Loai Abdallatif 
wrote:

> Dear Colleagues
> I have HR application deployed on two tomcat workers , but when I restart
> the tomcat instance or restart the OS , then the application failed to
> start and just works if I re-deploy the application again , please advise
>

What is the error you are getting? Please provide some details, otherwise
it will be very hard to help you out here.

>

Re: tomcat8/java8 "broken pipe" error

[Suvendu Sekhar Mondal]

On Tue, Jun 5, 2018, 8:34 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Holly,
>
> On 6/5/18 7:23 AM, Lund, Holly (CONTR) wrote:
> >
> > Continuously receiving the below error after upgrade to Tomcat
> > 8.0.43 /java 1.8.0_162/Apache 2.4.25 from tomcat6/java6/apache2.2
> > on solaris 10 OS
> >
> > This only happens under load
> >
> >
> > 29-May-2018 11:30:22.677 WARNING [commons-pool-EvictionTimer]
> > org.apache.tomcat.dbcp.dbcp2.SwallowedExceptionLogger.onSwallowE
> > xception An internal object pool swallowed an Exception.
> > java.sql.SQLRecoverableException: IO Error: Broken pipe (Write
> > failed) at
> > oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:682) at
> > oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:7
> 11)
> >
> >
> at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:385)
> > at
> > oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension
> .java:30)
> >
> >
> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:558)
> > at
> > org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:38)
> >
> >
> at
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Poolab
> leConnectionFactory.java:255)
> > at
> > org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObje
> ctPool.java:888)
> >
> >
> at
> org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureIdle(GenericOb
> jectPool.java:952)
> > at
> > org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureMinIdle(Gene
> ricObjectPool.java:931)
> >
> >
> at
> org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$Evictor.run(Base
> GenericObjectPool.java:1047)
> > at java.util.TimerThread.mainLoop(Timer.java:555) at
> > java.util.TimerThread.run(Timer.java:505) Caused by:
> > java.net.SocketException: Broken pipe (Write failed) at
> > java.net.SocketOutputStream.socketWrite0(Native Method) at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> >
> >
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > at oracle.net.ns.DataPacket.send(DataPacket.java:209) at
> > oracle.net.ns.NetOutputStream.write(NetOutputStream.java:180) at
> > oracle.net.ns.NetOutputStream.write(NetOutputStream.java:136) at
> > oracle.net.ano.AnoComm.a(Unknown Source) at
> > oracle.net.ano.Ano.negotiation(Unknown Source) at
> > oracle.net.ns.NSProtocol.connect(NSProtocol.java:292) at
> > oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
> > at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
> > ... 12 more
> >
> >
> > It appears that tomcat is disconnecting from the database after 60
> > seconds  (from Oracle logs)
> >
> > Context.xml
> >
> >
> >  > type="javax.sql.DataSource"
> > driverClassName="oracle.jdbc.OracleDriver"
> > url="jdbc:oracle:thin:@//averna.doe.gov:11900/ataaps.averna.doe.gov"
> >
> >
> username="xxx" password="xx" maxTotal="-1"   maxIdle="100"
> minIdle="5"
> > maxWaitMillis="30" removeAbandonedOnMaintenance="true"
> > removeAbandonedTimeout="300" logAbandoned="false"
> > testOnBorrow="true" testOnReturn="false"
> > timeBetweenEvictionRunsMillis="30"
> > minEvictableIdleTimeMillis="12" defaultAutoCommit="true"
> > initialSize="5" testWhileIdle="false" numTestsPerEvictionRun="5"
> > validationQuery="SELECT 1 FROM dual"/>
>
> I would check the settings on any firewall you have between the
> application and the database. Perhaps there is a 60-second
> connection-time-limit being imposed there?
>
> Also perhaps check to see if the database has a connection-limit for
> the user you are using. maxTotal="-1" seems ... like a bad idea.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlsWplwACgkQHPApP6U8
> pFjrhA/9FnQlYQ6yIyZhI+VJtnr+tO9JZJCX+kibFM9v9kLMKMhCDvoFvnuiJmDc
> nDlC2Al3uGqF8Py51CN8mkboZjqNORsr8yQzGpliLZTAyKkGLNq56tbejEwghLNF
> 6+g8HZDasz8V+8Yk0QL9QipNTW0+OinZORX38Bipvl6s7PtgYsSa1YlbEKpoO3gp
> ePpE+Iy8CePy3/uyx7UhL642ANE5fjh5pjP0q9DKbYC6dwH/vrYeRWoEqMnlm5oe
> MN3HvU6S2eIDil4XJ0nHPvGeIvfIS1fABu4JllOsWCobuzTVTFXlXXkHM5zfTf69
> Q3hnlO5HGR0jQPtJzpg9eSQZMFEh77xIyZ7lugyHweWGgyMBDar1fdo/wCuTCTdr
> X2Ec+NpY5MzZCfAWyJk/QVKN+a8ST8YQ+5FLfGcBqUVpxR6Od6Z05ZtHtw1iPcHE
> YRKMJLoV+Zf9dAruiS5HYuqC0fqi+k+mqtRKXmpFzLTyzAf29V+eXPL61DjSR7nh
> SvzLNPgSQssW0SYkTPItZBMC2Yq6hRnHLEXMBrAiQfNqeGOXEcUVOj/HwNc9qtRe
> DpcQSJ9qv3P2kbsDmQptz/7nn44OEQ1OtybG7OUIlsYdshnReL0WT9hhN2S+yiLO
> 1IkKJxSfyWaHa/W60c4lQ0kkFydfIFuzzpcpIJ1p18bMkSo6ffc=
> =OWPV
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org


Holly,

That error generally indicates a connection failure. App was 

Does Tomcat supports serialize session access?

[Suvendu Sekhar Mondal]

Hello Everyone,

We recently migrate our apps from Websphere(WAS) to Tomcat. We are
using httpd 2.4.10 in front of multiple Tomcat 7.0.55.

There is one feature of WAS that I am interested about and I am
looking for it everywhere in Tomcat world but could not found any
source. Feature called "serialize session access". If you turn that
"on", then it prohibits concurrent session access in a given JVM. It
ensures that only one request can access the session data at a time.

My questions are:
Is there any similar feature exists in Tomcat?
Does Tomcat works like that way - by default?
Is there any setting in Tomcat web container which can make processing
synchronized?


WAS feature: 
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/ae/tprs_serializing_access.html

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat 7 and 8 version memory setting

[Suvendu Sekhar Mondal]

Ramesh,

On Thu, Apr 26, 2018 at 11:45 AM, Naga Ramesh  wrote:
> Team,
>
>
>
> I have configured the tomcat7 & 8 versions on our AWS environment, but how
> much memory need to give min and max for each tomcat.  ( max to max how much
> need to give the max vlaue)
>
>
>
> Server:  AWS linux
>
> RAM: 128GB
>
>
>
> I have given min:4096M  and max:32768M
>

IMO, without analyzing the application's heap usage pattern, it's very
hard to speculate what should be the ideal heap size for any
application. If you set too low values, you might ended up seeing
frequent GCs and in worst case - OutOfMemory. If you go for a very
high value, and your application's requirement is much lesser than
that, then you have lots of unused memory(which you can use in
different purpose). As you have already setup a min/max value, I will
strongly suggest you to look closely at the heap usage for at least
one week. You can get that information in GC log(hopefully you are
logging it). After that you can set min/max heap size to a value which
is 20-30% more than your max heap usage. That will give your app some
head room so that in emergency situation(like very large request
processing or sudden spike in user traffic) your app behaves normally.

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is LDAP connection failing?

[Suvendu Sekhar Mondal]

On Wed, Apr 11, 2018, 3:00 PM Felix Schumacher <
felix.schumac...@internetallee.de> wrote:

> Am 05.04.2018 15:32, schrieb Suvendu Sekhar Mondal:
> > Hello Everyone,
> >
> > Recently in one of our environments I am seeing following log in
> > Catalina.out. It seems that LDAP connection is failing. This issue is
> > sporadic and goes away with Tomcat recycle.
> >
> > One interesting thing is "localhost:389" part. I could not find out
> > any configuration related to that. It could happen that I am not
> > looking at the correct place.
> >
> > We have 200+ JVMs out there which were starting up simultaneously but
> > this happens for some of them sporadically. I suspect that some race
> > condition might be causing this failure but could not found any
> > evidence so far. Can someone please suggest how can I identify what is
> > failing? and why it is failing?
>
> It would be nice to include the version of tomcat you are using.
> (I am guessing it is something like 7.0.55 as the source code matches
> the line
> numbers in the stacktrace)
>

Felix,

Sorry, I should have given that info upfront. You are correct. I'm using
7.0.55.

If it is this version, then the message will be generated, when your
> ldap server
> configured by connectionURL is not reachable on startup. Tomcat will try
> to
> connect to the ldap server configured by alternateURL. It seems to me,
> that
> you have not configured one (again guessing, as you didn't give
> configuration
> details). In that case the jre is using localhost:389 and as there is no
> LDAP server listening you get the exception.
>

You are right. We don't have any alternate URL configured. So, work around
we are using is to start those JVMs in batches. We are working on to tune
LDAP as well as to get alternate URL.

Thank you for the lead. Appreciate it! :)

>
> Regards,
>   Felix
> >
> > Thanks!
> > Suvendu
> >
> > Stack trace:
> > 2018-04-02 20:34:27,293 INFO org.apache.catalina.startup.HostConfig -
> > Deploying web application directory D:\xxx\webapps\ROOT
> > 2018-04-02 20:34:33,341 SEVERE org.apache.catalina.realm.CombinedRealm
> > - Failed to start "org.apache.catalina.realm.JNDIRealm/1.0" realm
> > org.apache.catalina.LifecycleException: Failed to start component
> > [Realm[JNDIRealm]]
> >  at
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
> >  at
> >
> org.apache.catalina.realm.CombinedRealm.startInternal(CombinedRealm.java:201)
> >  at
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> >  at
> >
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5373)
> >  at
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> >  at
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
> >  at
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
> >  at
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
> >  at
> >
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247)
> >  at
> >
> org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898)
> >  at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >  at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >  at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >  at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.catalina.LifecycleException: Exception opening
> > directory server connection
> >  at
> > org.apache.catalina.realm.JNDIRealm.startInternal(JNDIRealm.java:2191)
> >  at
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> >  ... 14 more
> > Caused by: javax.naming.CommunicationException: localhost:389 [Root
> > exception is java.net.ConnectException: Connection refused: connect]
> >  at com.sun.jndi.ldap.Connection.(Connection.java:216)
> >  at com.sun.jndi.ldap.LdapClient.(LdapClient.java:137)
> >  at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614)
> >  at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746)
> >  at com.sun.jndi.ldap.LdapCtx.(LdapCtx.java:319)
> >  at
> >
> com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:70)
> >

Is LDAP connection failing?

[Suvendu Sekhar Mondal]

Hello Everyone,

Recently in one of our environments I am seeing following log in
Catalina.out. It seems that LDAP connection is failing. This issue is
sporadic and goes away with Tomcat recycle.

One interesting thing is "localhost:389" part. I could not find out
any configuration related to that. It could happen that I am not
looking at the correct place.

We have 200+ JVMs out there which were starting up simultaneously but
this happens for some of them sporadically. I suspect that some race
condition might be causing this failure but could not found any
evidence so far. Can someone please suggest how can I identify what is
failing? and why it is failing?

Thanks!
Suvendu

Stack trace:
2018-04-02 20:34:27,293 INFO org.apache.catalina.startup.HostConfig -
Deploying web application directory D:\xxx\webapps\ROOT
2018-04-02 20:34:33,341 SEVERE org.apache.catalina.realm.CombinedRealm
- Failed to start "org.apache.catalina.realm.JNDIRealm/1.0" realm
org.apache.catalina.LifecycleException: Failed to start component
[Realm[JNDIRealm]]
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
 at 
org.apache.catalina.realm.CombinedRealm.startInternal(CombinedRealm.java:201)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5373)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
 at 
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1247)
 at 
org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1898)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Exception opening
directory server connection
 at org.apache.catalina.realm.JNDIRealm.startInternal(JNDIRealm.java:2191)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
 ... 14 more
Caused by: javax.naming.CommunicationException: localhost:389 [Root
exception is java.net.ConnectException: Connection refused: connect]
 at com.sun.jndi.ldap.Connection.(Connection.java:216)
 at com.sun.jndi.ldap.LdapClient.(LdapClient.java:137)
 at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614)
 at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746)
 at com.sun.jndi.ldap.LdapCtx.(LdapCtx.java:319)
 at 
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:70)
 at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
 at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
 at javax.naming.InitialContext.init(InitialContext.java:244)
 at javax.naming.InitialContext.(InitialContext.java:216)
 at 
javax.naming.directory.InitialDirContext.(InitialDirContext.java:101)
 at org.apache.catalina.realm.JNDIRealm.open(JNDIRealm.java:2108)
 at org.apache.catalina.realm.JNDIRealm.startInternal(JNDIRealm.java:2189)
 ... 15 more
Caused by: java.net.ConnectException: Connection refused: connect
 at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
 at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
 at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
 at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
 at java.net.Socket.connect(Socket.java:589)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.sun.jndi.ldap.Connection.createSocket(Connection.java:350)
 at com.sun.jndi.ldap.Connection.(Connection.java:203)
 ... 27 more

2018-04-02 20:34:35,059 INFO org.apache.catalina.startup.HostConfig -
Deployment of web application directory D:\xxx\webapps\ROOT has
finished in 7,766 ms
2018-04-02 20:34:35,075 INFO
org.apache.coyote.http11.Http11AprProtocol - Starting ProtocolHandler
["http-apr-18110"]
2018-04-02 20:34:35,091 INFO org.apache.coyote.ajp.AjpAprProtocol -
Starting ProtocolHandler ["ajp-apr-18111"]

Re: Instances of org.apache.coyote.RequestInfo accumulate in RequestGroupInfo.processors and eventually result in OOME

[Suvendu Sekhar Mondal]

On Wed, Mar 14, 2018 at 2:19 AM, Industrious  wrote:
> Hello, Mark,
>
> Thanks for your attention.
>
> Could you take a look at the class histogram from today's OOME heap dump?
> Maybe it could provide some details.
> I see a spike in CPU usage at the approximate time the dump was
> generated but that might be caused by the garbage collector's futile
> attempt to free up memory.

That's correct. Aggressive GCs causes that.

> Class Name |   Objects |
> Shallow Heap |  Retained Heap
> ---
>|   |
>|
> org.apache.coyote.RequestInfo  | 1 118 |
> 98 384 | >= 169 690 688
> org.apache.coyote.Request  | 1 118 |
>187 824 | >= 169 564 168
> byte[] |70 337 |
> 120 141 712 | >= 120 141 712


>From your problem description it seems that you are having a slow leak
which is crashing your Tomcat after few days. Are those RequestInfo
objects are of same size? or, some of them are large whereas rest of
them are small in size? Same question goes for byte arrays. From
output format it seems that you are using MAT. I will suggest to check
"path to gc roots" for those objects first. That will tell you who is
keeping them alive. Also please check the thread(s) associated with
those Byte arrays and RequestInfo objects. That will tell you if any
application thread is involved or not.

> org.apache.tomcat.util.net.SecureNioChannel|   985 |
> 47 280 |  >= 59 649 592
> char[] |   128 612 |
> 55 092 504 |  >= 55 092 504
> org.apache.tomcat.util.buf.CharChunk   |89 026 |
>  4 273 248 |  >= 41 134 168
> java.nio.HeapByteBuffer| 4 256 |
>204 288 |  >= 33 834 864
> org.apache.tomcat.util.net.NioEndpoint$NioBufferHandler|   985 |
> 23 640 |  >= 33 482 120
> org.apache.catalina.connector.Request  | 1 118 |
>187 824 |  >= 33 452 000
> org.apache.catalina.connector.Response | 1 118 |
> 71 552 |  >= 28 233 912
> org.apache.catalina.connector.OutputBuffer | 1 118 |
> 89 440 |  >= 27 898 496
> org.apache.catalina.connector.InputBuffer  | 1 118 |
> 80 496 |  >= 27 270 448
> sun.security.ssl.SSLEngineImpl |   985 |
>133 960 |  >= 25 596 024
> sun.security.ssl.EngineInputRecord |   985 |
> 63 040 |  >= 20 648 288
> org.apache.tomcat.util.buf.ByteChunk   |99 093 |
>  4 756 464 |  >= 15 422 384
> java.lang.String   |   108 196 |
>  2 596 704 |  >= 14 737 456
> org.apache.tomcat.util.buf.MessageBytes|84 554 |
>  4 058 592 |  >= 12 960 440
> java.util.HashMap$Node[]   | 9 139 |
>  1 156 352 |  >= 12 864 216
> java.util.HashMap  |10 997 |
>527 856 |  >= 12 817 352
> java.util.HashMap$Node |56 583 |
>  1 810 656 |  >= 11 484 248
> org.apache.catalina.loader.WebappClassLoader   | 2 |
>272 |  >= 10 199 128
> org.apache.coyote.http11.InternalNioOutputBuffer   | 1 118 |
> 89 440 |   >= 9 811 568
> java.util.concurrent.ConcurrentHashMap | 3 823 |
>244 672 |   >= 9 646 384
> java.util.concurrent.ConcurrentHashMap$Node[]  | 1 295 |
>260 664 |   >= 9 404 616
> java.lang.Class| 9 901 |
> 85 176 |   >= 9 233 664
> java.util.concurrent.ConcurrentHashMap$Node|15 554 |
>497 728 |   >= 9 111 176
> org.apache.tomcat.util.http.MimeHeaders| 2 236 |
> 53 664 |   >= 7 119 880
> org.apache.tomcat.util.http.MimeHeaderField[]  | 2 236 |
>141 248 |   >= 7 066 208
> org.apache.tomcat.util.http.MimeHeaderField|20 201 |
>484 824 |   >= 6 924 960
> org.apache.catalina.loader.ResourceEntry   | 5 133 |
>246 384 |   >= 5 414 616
> java.lang.Object[] |17 925 |
>  1 249 176 |   >= 5 046 960
> java.io.ByteArrayOutputStream  | 3 857 |
> 92 568 |   >= 4 603 096
> org.apache.catalina.webresources.JarResourceSet|46 |
>  3 312 |   >= 4 576 680
> java.text.SimpleDateFormat | 3 413 |
>218 432 |   >= 4 236 656
> java.text.SimpleDateFormat[]   | 1 126 |
> 36 032 |   >= 4 227 008
> sun.security.ssl.HandshakeHash |   985 |
> 39 400 |   >= 3 895 560
> Total: 36 of 9 892 

Re: Tool to analyze the core/heap dump

[Suvendu Sekhar Mondal]

On Tue, Feb 6, 2018 at 2:12 AM, Coty Sutherland  wrote:
> On Mon, Feb 5, 2018 at 3:16 PM, Igal @ Lucee.org  wrote:
>> On 2/5/2018 11:15 AM, Johan Compagner wrote:
>>>
>>> Jvisualvm that ships with java8 or yourkit (you can evaluate for some
>>> time)
>>>
>>> Op 5 feb. 2018 19:43 schreef "D, Dwarakesh (External)" <
>>> dwarakes...@xerox.com>:
>>>
 We have core and heap dump files generated from tomcat in our Solaris
 server. Is there any best tool to analyze those logs, please suggest on
 this.

 Thanks,
 Dwarakesh
>>
>> Try Eclipse MAT:
>> https://www.eclipse.org/mat/
>
> +1 for MAT, though it's a bit difficult if you're using IBM's JDK. For
> core dumps, ADB on Solaris works well.
>
MAT is great for heap dump analysis.
For core dumps you need to use gdb. Sometimes jstack and jmap helps too.

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Which Connector properties will be used in case of redirection?

[Suvendu Sekhar Mondal]

Hi Christopher,

On Wed, Jan 17, 2018 at 10:41 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Suvendu,
>
> Which version of Tomcat?
>

Tomcat version is 7.0.55

> On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote:
>> I am seeing one issue. Under high load sporadically one web
>> service call fails with: "java.net.SocketTimeoutException: Read
>> timed out" after 60 Sec mark.
>>
>> In our app, httpd routes requests to  then it get redirected
>> to 8889.
>
> Can you be very explicit about what you mean by "redirect", here?

I was talking about auto redirection from  to 8889 by Catalina.

>> Connector which listens on  has connectionTimeout=2 but the
>> one which listens to 8889 does not have any connectionTimeout set
>> explicitly. As per doc, default connectionTimeout value 60 Sec will
>> be set for connector which listens on 8889 - and that is playing a
>> role here. Do you think this assumption is correct?
>>
>> Here are the connector properties: > protocol="HTTP/1.1" connectionTimeout="2" redirectPort="8889"
>> />
>
> So you are probably expecting that all HTTP traffic to : will be
> redirected to HTTPS over :8889?
>

Yes.

> How long does the redirect take?
>

I do not know how to measure redirection time. Please let me know if
there any mechanism to trace it.

> Under heavy load, you might be running out of threads depending upon a
> lot of factors, such as what kind of protocol and endpoint are
> actually being used. It looks like you are using the APR connector
> (due to SSLCertificateFile) which has some positive attributes with
> request to scalability while NIO would probably be the best choice if
> possible.
>
> It's got terrible performance, though, unless you use the OpenSSL
> provider (which requires Tomcat 8.5 or later).
>
>> > SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true"
>> acceptCount="100" clientAuth="false" disableUploadTimeout="true"
>> enableLookups="false" keystoreFile="conf/.keystore"
>> maxHttpHeaderSize="8192" maxSavePostSize="-1" maxThreads="200"
>> minSpareThreads="20" protocol="HTTP/1.1" scheme="https"
>> secure="true" sslProtocol="TLS1.2"
>> compressableMimeTypes="text/html,text/xml,text/plain,application/json"
>>
>>
> compressionMinSize="2048"
>> compression="force" threadPriority="6" />
>
> When you say "high load", what kind of load are you talking about,
> specifically?
>
> You may simply have too much traffic for your existing hardware to
> handle. No amount of configuration can fix that.
> - -chris

We have 9 JVMs. As per current setting, per JVM max concurrency can
reach up-to 300(maxThreads="200"+acceptCount="100"). Issue was
reported when 2300 users on the system. So, I think it is not a case
of thread exhaustion but something else. Most possibly few JVMs were
under stress due to uneven load balancing(we have stickysession). This
issue just came to me and all resource usage data are gone. I have
asked for re-run so that I can analyze it properly. Let's see. :)

Back to my original question, in case of HTTP to HTTPS auto
redirection, Connector properties set for the later will be finally
applied to the request - is that a correct statement?

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Which Connector properties will be used in case of redirection?

[Suvendu Sekhar Mondal]

Hello Everyone,

I am seeing one issue. Under high load sporadically one web service
call fails with: "java.net.SocketTimeoutException: Read timed out"
after 60 Sec mark.

In our app, httpd routes requests to  then it get redirected to
8889. Connector which listens on  has connectionTimeout=2 but
the one which listens to 8889 does not have any connectionTimeout set
explicitly. As per doc, default connectionTimeout value 60 Sec will be
set for connector which listens on 8889 - and that is playing a role
here. Do you think this assumption is correct?

Here are the connector properties:




Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: GC allocation failure

[Suvendu Sekhar Mondal]

On Tue, Jan 9, 2018 at 10:53 AM, Leon Rosenberg
 wrote:
> On Mon, Jan 8, 2018 at 10:26 AM, Mark Thomas  wrote:
>
>> On 08/01/18 15:16, Christopher Schultz wrote:
>>
>> 
>>
>> >> Therefore, the first time that the GC runs, the process can take
>> >> longer. Also, the heap is more likely to be fragmented and require
>> >> a heap compaction. To avoid that, till now my strategy is to: -
>> >> Start application with the minimum heap size that application
>> >> requires - When the GC starts up, it runs frequently and
>> >> efficiently because the heap is small
>> >
>> > I think this is a reasonable expectation for someone who doesn't
>> > understand the Black Art of Garbage Collection, but I'm not sure it's
>> > actually true. I'm not claiming that I know any better than you do,
>> > but I suspect that the collector takes its parameters very seriously,
>> > and when you introduce artificial constraints (such as a smaller
>> > minimum heap size), the GC will attempt to respect those constraints.
>> > The reality is that those constraints are completely unnecessary; you
>> > have only imposed them because you think you know better than the GC
>> > algorithm.

Thank you for all your response.

Well, most of our clients are running on IBM J9 JVM and that is what
IBM recommends :):
https://www.ibm.com/support/knowledgecenter/SSYKE2_9.0.0/com.ibm.java.multiplatform.90.doc/diag/understanding/mm_heapsizing_initial.html

We have started moving our clients from WAS to Tomcat+ HotSpot JDK8
platform - that's why I am here, learning about it and throwing
questions :).

One thing about memory allocation by OS: if I setup different values
for initial and max, then after starting up the JVM, Windows
*reserves* the max amount of memory exclusively for the JVM. I get
that using Private Bytes counter. So that's why I believe there is no
chance of OOM at OS level. What I am more interested is about the cost
of heap expansion in HotSpot JVM.

>> Generally, the more memory available, the more efficient GC is. The
>> general rule is you can optimise for any two of the following at the
>> expense of the third:
>> - low pause time
>> - high throughout
>> - low memory usage
>>
>> It has been a few years since I listened to the experts talk about it
>> but a good rule of thumb used to be that you should size your heap 3-5
>> times bigger than the minimum heap used once the application memory
>> usages reaches steady state (i.e. the minimum value of the sawtooth on
>> the heap usage graph)
>>
>>
> Actually G1, which is very usable with java8 and default in jdk9, doesn't
> produce the sawtooth graph anymore.
> I also think the amount of memory has less influence on GC Performance in
> G1 or Shenandoah, but instead influence if they would perform a STW phase
> (which of course is also performance related, but differently).
> But I am not an expert either, so I might be wrong here.
>
> As for OP's original statement: "When the GC starts up, it runs frequently
> and
> efficiently because the heap is small", I don't think it is correct
> anymore, especially not for G1, as long as the object size is reasonable
> (not Humongous).
>
>
> Leon

Yes Leon, we are seeing that G1 is works best for our app. We have
some large objects and we can't reduce the size immediately. So we
have decided to increase G1 region size for the time being and
collecting dead Humongous objects during Young collections.


Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: GC allocation failure

[Suvendu Sekhar Mondal]

On Jan 4, 2018 11:14 PM, "Rainer Jung"  wrote:

Am 04.01.2018 um 18:20 schrieb Christopher Schultz:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Ambica,
>
> On 1/4/18 11:17 AM, Sanka, Ambica wrote:
>
>> I am seeing below highlighted errors in native_err logs in all my
>> tomcat applications. I also increased memory for the VM from 4GB to
>> 8GB. Still seeing those. When do we get that errors? I am reading
>> online that when program asks for memory and java cannot give,
>> that's when we see them. Please suggest. Java HotSpot(TM) 64-Bit
>> Server VM (25.20-b23) for linux-amd64 JRE (1.8.0_20-b26), built on
>> Jul 30 2014 13:13:52 by "java_re" with gcc 4.3.0 20080428 (Red Hat
>> 4.3.0-8) Memory: 4k page, physical 8061572k(2564740k free), swap
>> 4063228k(4063228k free)
>>
>> CommandLine flags: -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/opt/apache/ancillariesmonitoring/logs/
>> -XX:InitialHeapSize=128985152 -XX:MaxHeapSize=268435456
>> -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers
>> -XX:+UseCompressedOops -XX:+UseParallelGC
>>
>
> Others have commented on those messages you received, but nobody
> mentioned your heap configuration. In the above command-line
> arguments, you have specified both the minimum and maximum heap
> memory. You have expressed those values in bytes which makes it
> somewhat hard to read what they actually are, but this is what you
>

I *think* the JVM top line in GC output always shows bytes, even if you
were using other units in the original switches.


I agree.


have in readable units:
>
> - -XX:InitialHeapSize=128M -XX:MaxHeapSize=256M
>

but yes, that is a valid point!


So you aren't using an 8GiB heap. You aren't even using a 4GiB heap.
> You are using a 256 *megabyte* heap. If you really want an 8GiB heap,
> you'll need to set it properly in your command-line arguments.
>
> Note that setting the initial heap size to anything other than the
> maximum heap size just makes the JVM take longer to get the heap
> generations sized appropriately. For a long-running server process, I
> think it never makes any sense to set initial < max heap size. Always
> set them to the same value so that the heap itself does not have to be
> expanded/resized during heap allocations.


Christopher,

I really never found any explanation behind this "initial=max" heap size
theory until I saw your mail; although I see this type of configuration in
most of the places. It will be awesome if you can tell more about benefits
of this configuration.

I usually do not set initial and max heap size to same value because
garbage collection is delayed until the heap is full. Therefore, the first
time that the GC runs, the process can take longer. Also, the heap is more
likely to be fragmented and require a heap compaction. To avoid that, till
now my strategy is to:
- Start application with the minimum heap size that application requires
- When the GC starts up, it runs frequently and efficiently because the
heap is small
- When the heap is full of live objects, the GC compacts the heap. If
sufficient garbage is still not recovered or any of the other conditions
for heap expansion are met, the GC expands the heap.

Another thing, what if I know the server load varies a lot(from 10s in
night time to 1s during day time) during different time frame, does
"initial=max heap" apply for that situation also?

Please let me know what you think about it.

Thanks!
Suvendu

Re: GC allocation failure

[Suvendu Sekhar Mondal]

Ambica,

On Jan 4, 2018 9:47 PM, "Sanka, Ambica"  wrote:

I am seeing below highlighted errors in native_err logs in all my tomcat
applications. I also increased memory for the VM from 4GB to 8GB. Still
seeing those. When do we get that errors?


It is not an error. It is a very normal phenomenon for all Java based
application.

I am reading online that when program asks for memory and java cannot give,
that's when we see them. Please suggest.


That's true. Imagine this scenario: you have a warehouse where you keep
different types of stuff. Say you kept adding new stuffs daily. One day
you'll eventually run out of space. On that day you have two options:
 1. get rid off some old stuffs which are not needed and make room for the
new stuffs
2. Extend your old warehouse

Same thing happens when you run Java programs. What you are seeing in the
log that's called Garbage Collection(GC) and similar to opt#1. What you did
by increasing memory is like opt#2.

Again, GC activity is normal until that operation takes long time and
affect your application response time. I will suggest that please read
about Garbage Collection in Java. Google is your friend.

Thanks!
Suvendu

Java HotSpot(TM) 64-Bit Server VM (25.20-b23) for linux-amd64 JRE
(1.8.0_20-b26), built on Jul 30 2014 13:13:52 by "java_re" with gcc 4.3.0
20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 8061572k(2564740k free), swap 4063228k(4063228k
free)
CommandLine flags: -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/apache/ancillariesmonitoring/logs/
-XX:InitialHeapSize=128985152 -XX:MaxHeapSize=268435456 -XX:+PrintGC
-XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers
-XX:+UseCompressedOops -XX:+UseParallelGC
3.203: [GC (Allocation Failure)  31744K->6311K(121856K), 0.0097261 secs]
3.578: [GC (Allocation Failure)  38055K->12368K(121856K), 0.0089875 secs]
3.756: [GC (Allocation Failure)  44112K->19589K(121856K), 0.0100339 secs]
3.897: [GC (Allocation Failure)  51333K->25872K(153600K), 0.0092326 secs]
4.172: [GC (Allocation Failure)  89360K->38878K(153600K), 0.0152940 secs]
4.417: [GC (Allocation Failure)  102366K->50311K(148480K), 0.0148816 secs]
4.594: [GC (Allocation Failure)  95367K->49903K(151040K), 0.0197327 secs]
4.765: [GC (Allocation Failure)  94959K->50213K(148992K), 0.0149008 secs]
4.946: [GC (Allocation Failure)  96293K->52257K(150528K), 0.0172634 secs]
5.129: [GC (Allocation Failure)  98337K->53118K(151040K), 0.0139426 secs]
5.313: [GC (Allocation Failure)  102270K->53234K(152064K), 0.0122307 secs]
5.498: [GC (Allocation Failure)  102386K->53579K(153088K), 0.0166336 secs]
5.655: [GC (Allocation Failure)  104779K->54486K(153600K), 0.0161735 secs]
6.885: [GC (Allocation Failure)  105686K->51523K(153600K), 0.0123126 secs]

Thanks
Ambica.

Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?

[Suvendu Sekhar Mondal]

On Thu, Dec 28, 2017 at 3:36 PM, Olaf Kock  wrote:
>
> On 27.12.2017 23:16, Eric Robinson wrote:
>>>
>>> I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8
>>
>> I don't think so. This is a requirement of the software company whose
>> application solution we use. They are requiring us to move to tomcat 8 with
>> jdk 1.8. If we try to mix tomcat8 with jdk 1.6, supposedly we would have
>> problems. I guess all this is being driven by the need to switch to TLS 1.2.
>> I'm not sure if that would be a function of tomcat or java.
>
> As you were looking for the reason of the increased footprint, this would
> help you tremendously to get to the root cause, even if you don't intend to
> use it in production. Assume that a similar behavior already appears when
> you run tomcat6 with Java8 - in this case there might be little reason to
> continue to dig in tomcat, and assume that it's the Java implementation that
> caused your increased footprint.
>
> Tomcat 8 with Java 6 won't work (apart from the outdated implementation) as
> it requires at least Java7 (which is also outdated).
>
> And then there's yet another aspect: The memory footprint increased way
> below the price decrease for memory, so just adding this much memory could
> be filed away as a reasonable assumption within 7.5 years (JDK 1.6.21 was
> released July 2010, I'm assuming that this is the age of the hardware as
> well (because why would you have installed this version on newer hardware,
> when newer releases existed)
>
> I'm assuming that you have enough aspects to inspect - if you're really
> interested in finding the root cause, you'll need to come up with more
> specific measurements, e.g. profiler data, compare thread dumps and set up
> Tomcat 6 with Java 8 to have a third reference point.
>
> Olaf
>

How about turning on Native Memory Tracking for both Tomcat8+JDK1.8
and Tomcat6+JDK1.6 combination and compare the results? It will give
you details about internal memory consumption of the JVM. If you see
increase there, you will also able to find out which component's
memory footprint has increased. You can turn NMT on by adding
following JVM arguments:

-XX:+UnlockDiagnosticVMOptions
-XX:NativeMemoryTracking=summary

After turning on NMT and running Tomcat, when you see there is rise in
memory usage, use JCMD to dump native memory consumption details for
analysis. You can find more details about NMT here:
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

IMO, if you see there is not much difference between native memory
consumption of JDK1.6 and JDK1.8, then you need to focus on Tomcat.

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Are Symbol files of Tomcat DLLs publicly available?

[Suvendu Sekhar Mondal]

On Tue, Dec 5, 2017 at 4:19 PM, Mark Thomas <ma...@apache.org> wrote:
> On 05/12/17 06:47, Suvendu Sekhar Mondal wrote:
>> On Tue, Dec 5, 2017 at 1:28 AM, Mark Thomas <ma...@apache.org> wrote:
>>> On 04/12/17 11:12, Suvendu Sekhar Mondal wrote:
>>>> Hello Everyone,
>>>>
>>>> I am investigating a Tomcat crash. Actually, JRE crashed due to
>>>> "access violation" error. It created a Windows memory dump file. I am
>>>> trying to analyze it win WinDbg. Problem I am facing is that lots of
>>>> Symbols (of tomcat7, jvm, java, tcnative-1, nio DLLs) are not
>>>> available to me. As a result WinDbg is giving me a Stack filled up
>>>> with DLL names and HEX values.
>>>>
>>>> In order to get some of them - mostly JRE related, I have already
>>>> reached out to Java forum:
>>>> https://community.oracle.com/thread/4102753. No response so far :(.
>>>>
>>>> Can someone please tell me how can I get Symbol files of Tomcat DLLs
>>>> like tomcat7 and tcnative-1? Are they publicly available?
>>>
>>> Exactly which versions do you need?
>>
>> Sorry, Mark. I should have provide that information upfront. I am using:
>>
>> Tomcat 7.0.55
>> JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 
>> 1.8.0_92-b14)
>> OS version: Windows Server 2012 R2
>>
>>> I produced the binaries for the most recent of those and I probably
>>> still have the necessary files sat on a VM if they are recent(ish).
>>>
>>> Mark
>>
>> Yeah, I know it's not that recent. Still, if you have it please let me know.
>
> Those versions don't help. I need to know the exact version of
> tomcat7.exe and the tcnative-1.dll.
>
> Mark
>
>
>
>>
>> We tried to create PDB files for Java specific DLLs from OpenJDK, but
>> as you can see below, WinDbg rejected it.
>>
>> * Symbol Loading Error Summary **
>> Module nameError
>> tomcat7No header information available :
>> srv*c:\mss*http://msdl.microsoft.com/download/symbols
>> jvmSignature does not match :
>> srv*c:\mss*http://msdl.microsoft.com/download/symbols
>> tcnative-1 No header information available :
>> srv*c:\mss*http://msdl.microsoft.com/download/symbols
>>
>>
>> Thanks!
>> Suvendu
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Resurrecting this thread to provide some update.

First of all, sorry for this superb delayed response. I was out for vacation.

Good news is that we have found what is causing all those JVM crashes.
Client has created a huge Java class which contains 32000+ methods
(yes, we gave them a weapon and they have used it to hurt themselves
:( ). Our APM tool was trying to retrieve the stack frame for a method
which has something to do with those large number of methods in that
class. That is the point JVM crashed. We can recreate the issue with
OpenJDK, Java 8 and Java 9. It was found we can get around of it by
increasing stack size(-Xss) but that is not an optimal solution. We
have asked client to reduce the number of methods and also reported
this bug to Oracle. Let's see how it works.

And Happy New Year to all of you!!! :)

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Are Symbol files of Tomcat DLLs publicly available?

[Suvendu Sekhar Mondal]

On Tue, Dec 5, 2017 at 1:28 AM, Mark Thomas <ma...@apache.org> wrote:
> On 04/12/17 11:12, Suvendu Sekhar Mondal wrote:
>> Hello Everyone,
>>
>> I am investigating a Tomcat crash. Actually, JRE crashed due to
>> "access violation" error. It created a Windows memory dump file. I am
>> trying to analyze it win WinDbg. Problem I am facing is that lots of
>> Symbols (of tomcat7, jvm, java, tcnative-1, nio DLLs) are not
>> available to me. As a result WinDbg is giving me a Stack filled up
>> with DLL names and HEX values.
>>
>> In order to get some of them - mostly JRE related, I have already
>> reached out to Java forum:
>> https://community.oracle.com/thread/4102753. No response so far :(.
>>
>> Can someone please tell me how can I get Symbol files of Tomcat DLLs
>> like tomcat7 and tcnative-1? Are they publicly available?
>
> Exactly which versions do you need?

Sorry, Mark. I should have provide that information upfront. I am using:

Tomcat 7.0.55
JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
OS version: Windows Server 2012 R2

> I produced the binaries for the most recent of those and I probably
> still have the necessary files sat on a VM if they are recent(ish).
>
> Mark

Yeah, I know it's not that recent. Still, if you have it please let me know.

We tried to create PDB files for Java specific DLLs from OpenJDK, but
as you can see below, WinDbg rejected it.

* Symbol Loading Error Summary **
Module nameError
tomcat7No header information available :
srv*c:\mss*http://msdl.microsoft.com/download/symbols
jvmSignature does not match :
srv*c:\mss*http://msdl.microsoft.com/download/symbols
tcnative-1 No header information available :
srv*c:\mss*http://msdl.microsoft.com/download/symbols


Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Are Symbol files of Tomcat DLLs publicly available?

[Suvendu Sekhar Mondal]

Hello Everyone,

I am investigating a Tomcat crash. Actually, JRE crashed due to
"access violation" error. It created a Windows memory dump file. I am
trying to analyze it win WinDbg. Problem I am facing is that lots of
Symbols (of tomcat7, jvm, java, tcnative-1, nio DLLs) are not
available to me. As a result WinDbg is giving me a Stack filled up
with DLL names and HEX values.

In order to get some of them - mostly JRE related, I have already
reached out to Java forum:
https://community.oracle.com/thread/4102753. No response so far :(.

Can someone please tell me how can I get Symbol files of Tomcat DLLs
like tomcat7 and tcnative-1? Are they publicly available?

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8.5 Stuck Issue while trying to stop tomcat service

[Suvendu Sekhar Mondal]

Hi Athar,

On Nov 22, 2017 9:24 PM, "Khan, Athar (Contractor)" 
wrote:

Hi Team,



we have installed & using apache-tomcat-8.5.23 in our Pega 7.3 CRM
application. when we try to stop tomcat service from windows server-2012 R2
system tray GUI, It gets stuck & never gets stopped unless we kill the
service by task manager.

We are providing below the tomcat configuration details along with windows
server 2012R2. Could you please help us in providing the solution on this
issue.





*apache-tomcat-8.5.23 :*



*Java Virtual Machine:  *D:\Program Files\Java\jdk1.8.0_144\jre\
bin\server\jvm.dll



*Java classpath: * D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23\bin\bootstrap.jar;D:\Program
Files\apache-tomcat-8.5.23-windows-x64\apache-tomcat-8.5.
23\bin\tomcat-juli.jar



*Java Options:*



-Dcatalina.home=D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23

-Dcatalina.base=D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23

-Djava.endorsed.dirs=D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23\endorsed

-Djava.io.tmpdir=D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23\temp

-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

-Djava.util.logging.config.file=D:\Program Files\apache-tomcat-8.5.23-
windows-x64\apache-tomcat-8.5.23\conf\logging.properties

-Duser.home=E:\PEGA\Config

-Dpega.tmpdir=E:\PEGA\Temp

-Dpega.logdir=E:\PEGA\Logs

-verbose:gc

-Xloggc:E:\PEGA\TomcatLogs/gc.log

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-XX:+UseConcMarkSweepGC

-Dcom.sun.management.jmxremote.port=8056

-Dcom.sun.management.jmxremote.ssl=false

-Dcom.sun.management.jmxremote.authenticate=false



*Initial memory pool: *12288 MB

*Initial memory pool: *28672 MB





*Windows Server 2012R2:*









*Thanks*

*Athar Khan*



How long it stuck in that stage? Did you attached any error screenshots?
It's not visible. If possible please post the error message.

I will suggest you to take multiple thread dumps of the Tomcat process and
look what's going on there. Please don't send them as attachment as this
list might strip it off. You can upload it somewhere and post the link here.

Thanks!
Suvendu

Re: Question - Tomcat Memory

[Suvendu Sekhar Mondal]

On Nov 17, 2017 12:18 AM, "Ramya Elineni" 
wrote:

The "Initial memory pool" and "maximum memory pool" are the two
configurations under Tomcat. I couldn't find any detailed explanation of
how Tomcat uses these settings.


Those settings are equivalent to Xms and Xmx values respectively.

I

request you to please provide me that informaiotn so that I can have the
appropriate values set on a production environment.


Ideally you should put your app under load testing, measure heap usage and
after that you can set min/max heap. If you have lots of available RAM,
then you can bumped up  the max size by few gigs. But setting arbitrary
values without knowing the heap usage pattern can cause unwanted trouble.

We

are encountering issues where Tomcat is running on the higher end of the
maximum memory pool configuration after about 25 days of its last restart.
Thanks.


What problem(s) are you facing? Long pauses or crashes? Please look at the
GC log, if you are printing it already. Otherwise please use visualvm or
jconsole to get the memory usage details.

Re: TomCat service is running but not responding

[Suvendu Sekhar Mondal]

Darin,

On Oct 20, 2017 8:48 PM, "James H. H. Lampert" 
wrote:

On 10/20/17, 7:50 AM, André Warnier (tomcat) wrote:
. . .

Then also tell a bit more about the "stop responding" aspect. What
> exactly happens when you try to access tomcat, in the browser ?
> Are you not getting any answer at all, no mater how long you wait ?
> Are you gettin an error page ?
> Are you getting a blank page ?
>

And are there any messages (especially Java stack traces) in the
"catalina.out" log file that might indicate that something has gone
horribly wrong?

--
JHHL


In addition to the answer of the above questions, I will request you to
take at least 3 thread dumps 5Sec apart, when you think Tomcat service has
"stopped responding". Also look for OutOfMemory condition. Please post
those dumps in this list.

Thanks!
Suvendu

Re: Performance settings for Multiple Hosts

[Suvendu Sekhar Mondal]

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:
> I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
> WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
> dedicated to JSPWiki.  The other 8 are running variations of the same custom
> application with around 10-15 individual webapps each.
>
> When I am running 7 of the 10 hosts, performance is great.  I get JSP
> response time under a second.  But when I add just a couple of more of the
> hosts, my page response time on all of the apps goes from an acceptable
> under a second to horrible at around 15-20 seconds per page.  It doesn't
> seem to be a specific host causing the problem.  Reducing overall hosts in
> any order makes the problem go away.
>
> I've looked at the the processor utilization during the good times and slow
> times, and don't see a significant difference.  I have 16GB of memory, and
> it consistently shows about 35% utilization.  I also checked mySQL response
> time, and the per-query SQL response does not vary.  So it doesn't appear to
> be a db problem.
>
> I suspect there is some TC configuration parameter such as heap, etc that I
> need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
> which one, and I don't know a formula to use to figure out what to set it
> to.  So I just need a little education.  What tools can I use to help me
> figure out what is going south slowing everything to a crawl when the extra
> hosts come online?  And what parameters should I be looking at (and how
> should I calculate the proper values based on number of hosts)?
>
> Suggestions?
>
Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 7 giving java.lang.OutOfMemoryError: unable to create new native thread Exception in Catalina.out after upgrading to RHEL 7.4

[Suvendu Sekhar Mondal]

Radhika,

On Tue, Sep 19, 2017 at 2:21 PM, Peddi, Radhika (Radhika)
 wrote:
> Hi,
>
> We have upgraded RHEL 7.2 to RHEL 7.4. After upgrade when we are running 
> performance testing of our application we are seeing below error in 
> catalina.out.
>
> java.lang.OutOfMemoryError: unable to create new native thread Exception in 
> Catalina.out after upgrading to RHEL 7.4
>

This simply indicates that JVM was trying to create a new thread and
OS can't create any new thread simply because native memory space was
exhausted. This limit is very much platform dependent.

Is your app opening too many threads? On my Windows 10 laptop with
32GB RAM and JDK 1.8, I can open 53+ threads before run into this
OOM problem. You can find it by monitoring Tomcat instance. So far I
ran into this problem one time - couple of years ago. We got away
after adding some memory and by reducing thread stack size.

Another thing can happen, you might have lost some OS setting(which
was used to bump up the default OS limit) while upgrading from one
version to another; which was preventing this issue from occurring so
far - just speculating. And this is not a Tomcat's problem. :)

> Attached is the thread dump.
>
> We are using Tomcat 7.0.69.
>
> As per REDHAT we have increased tomcat5 user which used to run Tomcat process 
> in /etc/security/limits.d/20-nproc.conf like below
>
> * soft nproc 4096
> root soft nproc unlimited
> tomcat5 soft nproc 8192
>
> All the processes are used by Tomcat. If we increase the value to 16K or 32K 
> all threads are consumed and they are in Waiting(Parking) state.
>
> "http-bio-8443-exec-56" #973 daemon prio=5 os_prio=0 tid=0x7efc8c029800 
> nid=0x10e7b waiting on condition [0x7ef9bc7ee000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0003c454c008> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
> at 
> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>
> "http-bio-8443-exec-55" #972 daemon prio=5 os_prio=0 tid=0x7efc0401e000 
> nid=0x10e7a waiting on condition [0x7ef9bc82f000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0003c454c008> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
> at 
> org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)
>
> They are not getting release. After some time Tomcat getting stopped 
> automatically. You may need the output of below from our setup.
>
> [root@iqsvtk05 ~]# rpm -qif 
> /opt/coreservices/tomcat-7.0.69/logs/
> Name : tomcat
> Version : 7.0.69
> Release : AV10
> Architecture: x86_64
> Install Date: Fri 18 Aug 2017 01:41:54 AM MDT
> Group : System Environment/Daemons
> Size : 8468465
> License : Apache License, Version 2.0
> Signature : (none)
> Source RPM : tomcat-7.0.69-AV10.src.rpm
> Build Date : Fri 24 Jun 2016 01:59:37 AM MDT
> Build Host : puiqx3650dev03.apac.avaya.com
> Relocations : /usr/share
> Packager : Avaya, Inc.

Re: Performance issue 8.5.20 (metaspace related?)

[Suvendu Sekhar Mondal]

> Profiling the code I have been able to find the cause of the big metaspace 
> garbage…
> Due to a bug, we were not caching remote interfaces when connecting to jboss 
> from the web sites. Other client kind was ok.
> Fixed this problem today, it’s a few hours that the system is running fine. 
> It’s fast (faster than before even with a fresh tomcat); metaspace says low 
> and everything is fine.
>

That's a very good news!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance issue 8.5.20 (metaspace related?)

[Suvendu Sekhar Mondal]

Andrea,

>> Sometimes Full GC were able to clean up some space from Metaspace but
>> only as part of a final last ditch collection effort:
>>
>> 43618.504: [Full GC (Last ditch collection)  1386M->250M(20G), 1.6455823 
>> secs]
>>   [Eden: 0.0B(6408.0M)->0.0B(6408.0M) Survivors: 0.0B->0.0B Heap:
>> 1386.7M(20.0G)->250.5M(20.0G)], [Metaspace:
>> 1646471K->163253K(1843200K)]
>> [Times: user=2.23 sys=0.10, real=1.65 secs]
>> 49034.392: [Full GC (Last ditch collection)  1491M->347M(20G), 1.9965534 
>> secs]
>>   [Eden: 0.0B(5600.0M)->0.0B(5600.0M) Survivors: 0.0B->0.0B Heap:
>> 1491.1M(20.0G)->347.9M(20.0G)], [Metaspace:
>> 1660804K->156199K(2031616K)]
>> [Times: user=2.78 sys=0.10, real=2.00 secs]
>
> This is interesting because I see this pattern for metaspace gc…
>
> https://ibb.co/b2B9HQ
>
> there’s something that clears it but it seems it’s not the full gc. I don’t 
> think there’s something that happens in the app that causes that much 
> metaspace  space to become instantly dead. So I think the full gc is not 
> doing the metaspace cleaning the way it could do it. Maybe it’s a light clean…
>

I believe MinMetaspaceFreeRatio and MaxMetaspaceFreeRatio playing a
role there. Default values for them in JDK 1.8 is 40 and 70
respectively. You can read about it more here:
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/considerations.html

>> That's is the default value. Please see:
>> http://docs.oracle.com/javase/8/docs/technotes/guides/rmi/sunrmiproperties.html
>>  
>> 
>
> They must have changed it recently, it was 180k when I first checked that 
> option. Do you think it’s worth to increase it ? I don’t see a full gc every 
> hour…
>

I have seen that if you use G1 GC, then JVM completely ignores those
RMI DGC parameter values. If you use Parallel GC, then you will see
periodical Full GCs(System.gc) are getting triggered based on those
param values. I did not get chance to go in-depth about this behavior
of G1. As G1 is new and much more efficient than other collectors(I
know this statement can draw some arguments), probably it is cleaning
those RMI objects more efficiently.

I will suggest that if you have some budget, please get a decent APM
like AppDynamics, New Relic, Dynatrace to monitor your prod system
24x7x365. Trust me, you will be able to identify and solve this type
of sporadic slowness issue very quickly.

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance issue 8.5.20 (metaspace related?)

[Suvendu Sekhar Mondal]

Andrea,

On Tue, Aug 29, 2017 at 5:39 PM, Ing. Andrea Vettori
<a.vett...@b2bires.com> wrote:
>> On 29 Aug 2017, at 12:29, Suvendu Sekhar Mondal <suv3...@gmail.com> wrote:
>>
>> On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori
>> <a.vett...@b2bires.com> wrote:
>>> - with a fresh started tomcat instance, the time it takes is around 0,8 
>>> seconds. Most of the time is spent on the two RMI calls the task does.
>>> - with an instance that is running from some time, the time can reach 2/3 
>>> seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. 
>>> I.e. what slows down are the RMI calls.
>>
>> Sill question, do you mean RMI calls generating from Tomcat is getting
>> slower with time? or, JBoss is taking time to return the response?
>
> Thanks for your help.
> What I see is that the http requests are slower. Looking at the details of 
> that specific request (putting a product in the cart) using System.nanotime I 
> can see that most of the time is spent during the RMI calls. I’m pretty sure 
> it’s not jboss that is slower because doing other calls at the same time with 
> fresh tomcats or with a java client they're not slow.
>

I will still suggest to monitor JBoss closely because it is important
part of your workflow. Please check GC stats also.

>>
>>> - restarting the jvm fixes the issue
>>> - ***it seems*** but I’m still testing this since it seems there’s no 
>>> ‘meatspace gc trigger command available', that when Metaspace is garbage 
>>> collected, tomcat then performs like a fresh instance.
>>>
>>
>> Yes, no command to only garbage collect Metaspace area. Without
>> recycling Tomcat, you can trigger a Full GC either using VisualVM or
>> jcmd's GC.run command.
>
> I think that explicit GC is not doing a metaspace GC… that’s what I observed 
> starting the GC with the java console.
>

In Hotspot VM, Full GC will collect "dead" objects/classes from all
the regions(Young, Old, Metaspace/PermGen). In the provided GC log, I
can see that you had triggered Full GCs but Metaspace did not shrunk
much. That clearly indicates you had good number of classes that were
"in use":

48318.001: [Full GC (Heap Dump Initiated GC)  5232M->1298M(20G), 4.9784080 secs]
   [Eden: 2800.0M(3136.0M)->0.0B(9616.0M) Survivors: 120.0M->0.0B
Heap: 5232.7M(20.0G)->1298.9M(20.0G)], [Metaspace:
1419958K->1398648K(2621440K)]
 [Times: user=9.55 sys=0.09, real=4.98 secs]
48404.270: [Full GC (System.gc())  2825M->1301M(20G), 4.3863751 secs]
   [Eden: 1528.0M(9616.0M)->0.0B(9616.0M) Survivors: 0.0B->0.0B Heap:
2825.2M(20.0G)->1301.6M(20.0G)], [Metaspace:
1425628K->1425623K(2627584K)]
 [Times: user=9.09 sys=0.15, real=4.38 secs]

Sometimes Full GC were able to clean up some space from Metaspace but
only as part of a final last ditch collection effort:

43618.504: [Full GC (Last ditch collection)  1386M->250M(20G), 1.6455823 secs]
   [Eden: 0.0B(6408.0M)->0.0B(6408.0M) Survivors: 0.0B->0.0B Heap:
1386.7M(20.0G)->250.5M(20.0G)], [Metaspace:
1646471K->163253K(1843200K)]
 [Times: user=2.23 sys=0.10, real=1.65 secs]
49034.392: [Full GC (Last ditch collection)  1491M->347M(20G), 1.9965534 secs]
   [Eden: 0.0B(5600.0M)->0.0B(5600.0M) Survivors: 0.0B->0.0B Heap:
1491.1M(20.0G)->347.9M(20.0G)], [Metaspace:
1660804K->156199K(2031616K)]
 [Times: user=2.78 sys=0.10, real=2.00 secs]

>>
>>> Since we’re using more than one tomcat instance (2 in production for this 
>>> website, 1 for development) I can see that the issue is isolated to Tomcat 
>>> or the JVM/Host where it runs because other Tomcat instances behave well at 
>>> the same time one is slow. The same JBoss/Postgres backend is used by java 
>>> batches and fat clients and it does work well with consistent times.
>>>
>>> To clarify: the moment one production tomcat that is running from some time 
>>> finishes the task in 3 seconds, the development tomcat or a fresh started 
>>> production tomcat instance does the same task in less that one second. Note 
>>> that repeating the task gives always consistent results, i.e. the instance 
>>> is running from some time is always slow,  the fresh running instance is 
>>> always fast.
>>>
>>> Tomcat is running with these VM options:
>>>
>>> -Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360 
>>> -Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt 
>>> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 
>>> -Dcom.sun.management.jmxremote.ssl=false 
>>> -Dcom.sun.management.jmxremote.authenticate=false -XX:+PrintGCDet

Re: Performance issue 8.5.20 (metaspace related?)

[Suvendu Sekhar Mondal]

Hello,

On Tue, Aug 29, 2017 at 2:54 PM, Ing. Andrea Vettori
 wrote:
> Hello,
> W're running several instances of Tomcat 8.5.20 / JDK8.144 / CentOs7 on our 
> company for various web sites in many hosts. Recently I’m trying to 
> understand a performance problem we’re having on our e-commerce web site.
> The configuration is the following
>
> HAProxy   <—> 2x Tomcat 8.5.20 <—>  JBoss 5.1 EJB <—> Postgres 9.6
>
> Tomcat runs a web site built with Struts / Freemarker that does call JBoss 
> EJBs with RMI.
> Monitoring a specific task (putting a product on the cart) I see the 
> following :
>
> - with a fresh started tomcat instance, the time it takes is around 0,8 
> seconds. Most of the time is spent on the two RMI calls the task does.
> - with an instance that is running from some time, the time can reach 2/3 
> seconds; occasionally 5/6 seconds. Most time is still spent on RMI calls. 
> I.e. what slows down are the RMI calls.

Sill question, do you mean RMI calls generating from Tomcat is getting
slower with time? or, JBoss is taking time to return the response?

> - restarting the jvm fixes the issue
> - ***it seems*** but I’m still testing this since it seems there’s no 
> ‘meatspace gc trigger command available', that when Metaspace is garbage 
> collected, tomcat then performs like a fresh instance.
>

Yes, no command to only garbage collect Metaspace area. Without
recycling Tomcat, you can trigger a Full GC either using VisualVM or
jcmd's GC.run command.

> Since we’re using more than one tomcat instance (2 in production for this 
> website, 1 for development) I can see that the issue is isolated to Tomcat or 
> the JVM/Host where it runs because other Tomcat instances behave well at the 
> same time one is slow. The same JBoss/Postgres backend is used by java 
> batches and fat clients and it does work well with consistent times.
>
> To clarify: the moment one production tomcat that is running from some time 
> finishes the task in 3 seconds, the development tomcat or a fresh started 
> production tomcat instance does the same task in less that one second. Note 
> that repeating the task gives always consistent results, i.e. the instance is 
> running from some time is always slow,  the fresh running instance is always 
> fast.
>
> Tomcat is running with these VM options:
>
> -Xms20480m -Xmx20480m -Dsun.rmi.dgc.client.gcInterval=360 
> -Dsun.rmi.dgc.server.gcInterval=360 -Xloggc:/tmp/gc.txt 
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false -XX:+PrintGCDetails 
> -XX:+PrintGCTimeStamps -XX:+UseG1GC -XX:ReservedCodeCacheSize=1g 
> -XX:InitialCodeCacheSize=256m -XX:+UseHugeTLBFS -XX:MetaspaceSize=1g 
> -XX:MaxMetaspaceSize=2g
>
> Some of the options have been recently added (for example the increase in 
> code cache  size) but it seems they had no impact on the issue.
>

Is there any specific reason to trigger Full GC every
60min(-Dsun.rmi.dgc.client.gcInterval=360)? I think that's
overkill.


> Metaspace goes up to 1,6GB before being collected. Value after garbage 
> collect is around 200MB. Heap usage is variable, it usually stays under 10G 
> and is around 1G after garbage collect.
> CPU usage rarely goes over 10%. Loaded classes between 20k and 40k. Active 
> sessions around 100/120 for each instance.
>

That sounds like a healthy behavior to me. That means GC is occurring
and Metaspace is getting cleaned up. Why do you think Metaspace is the
problem?
>From the given data, my first impression is JVM is not having any
memory pressure. If you can post your gc.txt file to this group, I
would be happy to check it.

Another thing, is there any way you can measure the processing time in
JBoss for each RMI calls?

> Any help or direction to understand what’s causing this is greatly 
> appreciated.
> Thank you
> —
> Ing. Andrea Vettori
>

Thanks!
Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Refreshing webapps slows server

[Suvendu Sekhar Mondal]

On Aug 23, 2017 11:40 PM, "Caldarale, Charles R" 
wrote:

> From: Jerry Malcolm [mailto:techst...@malcolms.com]
> Subject: Refreshing webapps slows server

> I have a very weird situation.

Actually, it's fairly common.

> This is somewhat circumstantial.  But TC will run fine for days and
> never hits OutofMemory situations.  But as soon as I start replacing
> webapp jar files, things start going bad.  So it appears that the issue
> is caused by replacing jar files.

This sounds like a classic case of retaining references to now obsolete
classes or instances thereof.  Take a look at the Wiki:
https://wiki.apache.org/tomcat/FAQ/Memory
especially, the link to "classloaders are not being garbage collected" and
these:
https://wiki.apache.org/tomcat/OutOfMemory
https://wiki.apache.org/tomcat/MemoryLeakProtection

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you received
this in error, please contact the sender and delete the e-mail and its
attachments from all computers.


I agree with Chuck.

Jerry,
In this type of situation, if I were you, I would have tried to found the
root cause of OutOfMemory and fix it. Because if you are using same method
to apply your new code to the production server and if this problem
remains, there is high chance your prod server will also suffer from
OutOfMemory
- sooner or later. I will suggest you to examine those heap dumps using
your favourite tool. I am pretty sure you will get solid clue to solve this
problem.

Thanks!
Suvendu

Re: TomcatCon London - registration open

[Suvendu Sekhar Mondal]

Just wondering when we will have TomcatCon in India... :)

On Aug 22, 2017 10:21 PM, "Mark Thomas"  wrote:

> All,
>
> The Apache Tomcat PMC is delighted to announce that registration for
> TomcatCon London is now open.
>
> This one day conference will be held on Tuesday 26th September in
> central London and features talks from:
> - Mark Thomas (markt)
> - Rémy Maucherat (remm)
> - Jean-Frederic Clere (jfclere)
>
> Full details, including the schedule is available on the website:
> http://tomcat.apache.org/conference.html
>
> Registration is via EventBrite:
> https://www.eventbrite.com/e/tomcatcon-london-2017-tickets-36683639754
>
> We hope to see you there!
>
> Mark
> on behalf of the Apache Tomcat PMC
>
>
> This event is sponsored by Liferay, the robust java-based open source
> platform (5 million downloads) used to build sophisticated transactional
> websites and deliver them to web browsers and native mobile clients.
> Liferay, Inc provides enterprise support for the commercial release
> which is used by organisations such as Allianz, Volkswagen Group and the
> NHS.
>
> https://web.liferay.com/community/liferay-projects/liferay-portal/overview
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat server apparently bouncing up and down

[Suvendu Sekhar Mondal]

James,

Since you told the context is rather huge, have you checked gc times? A
long running full gc can block the machine completely resulting in the
up/down behaviour from outside. GC options depend on JVM version I use:

export JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC -verbose:GC
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log "

alternatively moskito would also show you avg response times and gcs ;-)


regards

Leon


I agree. Long GC pauses can cause this type of issue. I have seen somewhat
similar. Apache PING failed due to long GC pause and user got 503s.

One question, do you see evidence of Tomcat restart in logs which
correlates with alerts?

Re: Tomcat memory

[Suvendu Sekhar Mondal]

er). Below
>>> you
>>> are showing the startup parameters of the JVM. They are not the default
>>> settings. So where do these parameters come from, and do you understand
>>> them ? (if you don't, you first need help with Java, not with Tomcat)
>>>b) That is really the focus of this list. And we know that Tomcat, by
>>> itself, does not use 60GB or 10GB of memory, nor anywhere near it
>>>c) we know nothing about, but it is likely that it is something there,
>>> which causes Tomcat (and Java) to use all that memory
>>> And about (c), maybe it is normal that it/they use this gigantic amount
>>> of
>>> memory. Maybe one of the applications allocates some gigantic array of
>>> data
>>> over time. Maybe one of these applications is really badly programmed and
>>> causes enormous "memory leaks". How would we know ? Are these your
>>> applications ? If not, have you contacted the people responsible for
>>> these
>>> applications, and described your problem to them ?
>>> (because again, with 99& probability, that is where the problem is)
>>>
>>> Try to collect some more specific data, and when you have it, post it
>>> here, and maybe someone will have an idea.  But as it is, the only thing
>>> known so far is that on your system, the Java JVM which runs tomcat is
>>> using up a lot of memory, increasingly over time, and that it ends up
>>> crashing with an OOM.
>>> That is not enough information for us to be able to help you.
>>>
>>> If you are desperate and in a hurry, maybe you need first to find a Java
>>> JVM specialist, who can help you poinpoint the problem more specifically.
>>>
>>>
>>>
>>>
>>>> Regards,
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 9:46 AM, Fady Haikal <fadyhai...@gmail.com>
>>>> wrote:
>>>>
>>>> @Suvendu,
>>>>>
>>>>> I took a heap dump from Java VisualVM but honestly i didnt know how i
>>>>> should analyse it, please some help here
>>>>>
>>>>> also please find below the java configuration i used:
>>>>> -XX:PermSize=10240m
>>>>> -XX:MaxPermSize=10240m
>>>>> -XX:ReservedCodeCacheSize=512m
>>>>> -XX:+UseParNewGC
>>>>> -XX:+UseConcMarkSweepGC
>>>>> -XX:+CMSParallelRemarkEnabled
>>>>> -XX:TargetSurvivorRatio=90
>>>>>
>>>>> Initial and Max Memory Pool: 10MB
>>>>>
>>>>> Regards,
>>>>> Fady
>>>>>
>>>>>
>>>>> On Wed, Aug 16, 2017 at 5:46 PM, Suvendu Sekhar Mondal <
>>>>> suv3...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Fady,
>>>>>>
>>>>>>
>>>>>> On Aug 16, 2017 7:04 PM, "Fady Haikal" <fadyhai...@gmail.com> wrote:
>>>>>>
>>>>>> Dear Team,
>>>>>> I'm facing an issue that tomcat from task manager is consuming around
>>>>>> 60
>>>>>> GB
>>>>>> of memory while from Oracle Java Mission Control is showing maximum 10
>>>>>> GB
>>>>>> (Attached screenshots), and from time to time the server hang due to
>>>>>> insufficent memory.
>>>>>> can you please advise why the above is showing like that? and why the
>>>>>> memory of the server is not reducing?
>>>>>>
>>>>>> Tomcat Version: 8.0.30
>>>>>> JDK: 1.7.0_67
>>>>>>
>>>>>>
>>>>>> I couldn't find the screenshot. Anyway, the issue you mentioned is
>>>>>> very
>>>>>> interesting. Usually I saw some difference(in MB) between task manager
>>>>>> data
>>>>>> and JMC data...but not like this.  Can you please let us know about
>>>>>> the
>>>>>> Java Heap, Permgen and thread stack size configuration?
>>>>>>
>>>>>> For starter, I will suggest to take a heap dump and look into it. It
>>>>>> will
>>>>>> show you which objects are live and holding the memory. Also it will
>>>>>> be
>>>>>> awesome if you can enable Native Memory Tracking for your TC instance.
>>>>>>
>>>>>> Thanks!
>>>>>> Suvendu
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat memory

[Suvendu Sekhar Mondal]

Hi Fady,

On Thu, Aug 17, 2017 at 12:16 PM, Fady Haikal  wrote:
> @Suvendu,
> I took a heap dump from Java VisualVM but honestly i didnt know how i
> should analyse it, please some help here

Acquire the software:
André has already given you some pointer. My favorite is Eclipse
MAT[http://www.eclipse.org/mat]. :)
- Download the standalone version.
- To open a large dump, you need to have a server with sufficient
amount of memory. I assume you have such machine. :)
- Backup MemoryAnalyzer.ini. Then Remove everything inside it and add
following in two lines...make sure there is no space before and after
those args:
   -vmargs
   -Xmx14g
It will create a VM for MAT with 14GB heap. For me it opens up 6GB
heap in 5-8min. Adjust heap size as required.

Analysis:
You need to change Memory Look at the object size in 'Dominator Tree'
view(by looking at the Retained and Shallow Heap Size) and use 'path
to GC root' option to see what is keeping them alive. If MAT shows
that small number of objects are consuming most of the heap, then you
are lucky. If very small objects are filling up the heap then your job
will become tedious. There is one 'Leak Suspect' view but it may or
may not be accurate - depending on your application architecture.

I will strongly suggest that you go through some documentation and how
to use MAT stuffs so that you become more familiar with it :
http://www.eclipse.org/mat/documentation

There are lots of docs available over Internet to help you in analysis part.

> also please find below the java configuration i used:
> -XX:PermSize=10240m
> -XX:MaxPermSize=10240m
> -XX:ReservedCodeCacheSize=512m
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:TargetSurvivorRatio=90
>
> Initial and Max Memory Pool: 10MB
>
> Regards,
> Fady
>

So you have created a huge JVM. This is the high level formula I use
to predict approximate JVM process size:
JDK 1.7 JVM Process Size = [-Xmx] + [-XX:MaxPermSize] + [-Xss] *
number_of_threads + JIT’ed code + GC + Other Stuffs

In your case, from above parameter, (max)approximate size could be
10MB+10240MB+some more memory. Because I do not know how many
threads you have opened. By default I guess JDK 1.7 allocates 1MB
stack space per thread. Again, I will suggest to enable NMT if
possible. It tells us where most of the memory is being consumed which
you can not trace using heap dumps.

You are using CMS. I have seen that it can create problem if it is not
properly tuned. I hope you have came up with those GC params after
good amount of performance testing.

One last thing, I agree with André, most probably you are dealing with
JVM problem not Tomcat. Please keep us posted about your findings.

Thanks!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat memory

[Suvendu Sekhar Mondal]

Hi Fady,

On Aug 16, 2017 7:04 PM, "Fady Haikal"  wrote:

Dear Team,
I'm facing an issue that tomcat from task manager is consuming around 60 GB
of memory while from Oracle Java Mission Control is showing maximum 10 GB
(Attached screenshots), and from time to time the server hang due to
insufficent memory.
can you please advise why the above is showing like that? and why the
memory of the server is not reducing?

Tomcat Version: 8.0.30
JDK: 1.7.0_67


I couldn't find the screenshot. Anyway​, the issue you mentioned is very
interesting. Usually I saw some difference(in MB) between task manager data
and JMC data...but not like this.  Can you please let us know about the
Java Heap, Permgen and thread stack size configuration?

For starter, I will suggest to take a heap dump and look into it. It will
show you which objects are live and holding the memory. Also it will be
awesome if you can enable Native Memory Tracking for your TC instance.

Thanks!
Suvendu

Re: Tomcat 8 slow shutdown on Windows 10

[Suvendu Sekhar Mondal]

Lazar,

> I am observing slow shutdown of Tomcat 8 on Windows 10 - some 9 - 11
> seconds. I observed this first with Tomcat 8.5.16, but then checked it on
> older versions and I found the same from 8.5.11 onward. Before that it
> stops for less than a second.
>

>
> Has anybody experienced similar problems?
>

I recently started using Tomcat 8.0.20 on Windows 10 and it takes more
than 1.5-2min to stop. This is much slower than 7.0.55. I thought it's
only happening on my system until I saw your mail. :)

I can see that you have already shared some thread dumps. Let me
collect some data during this slowdown. Wanted to see if it shows the
same hotspot or not.

Thanks!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Web application jars gets re loaded causing permgen issue !

[Suvendu Sekhar Mondal]

Utkarsh,

On Jul 28, 2017 2:10 PM, "Utkarsh Dave"  wrote:

Hi Users,

I am running Tomcat 7.0.77 on a linux server with openJDK 1.7.0.131
I am working on an issue where within 2-3 days Tomcat goes out of memory
because of java.lang.OutOfMemoryError: PermGen space .
After monitoring permgen via jconsole and checking catalina logs we noticed
the information messages

Jul 12, 2017 11:27:02 AM org.apache.catalina.startup.HostConfig reload

INFO: Reloading context [/abc]

Jul 12, 2017 11:27:02 AM org.apache.catalina.core.StandardContext reload
INFO: Reloading Context with name [/abc] has started

I see web.xml within webapp/abc/WEB-INF has the same time stamp of that of
the information message.
This makes me deduce that since Tomcat sees the new time stamp on web.xml,
it reloads this web-app.
I, however, do not see the date/time stamp change on webapp/abc folder.
Meaning the web application does not get deployed but it getting reloaded
into Tomcat.


Do you have set "context reloadable=true" somewhere in your app context?
That happens when you set that. But as far as I know, that process should
clear all old references first and then reload all classes and libraries.
Definitely some of the old references remains in the memory which is
gradually filling Permgen.

How can i debug this further i find who is responsible in changing web.xml
(If my analysis above is correct).
Should adding a stack trace within tomcat code (from where "INFO: Reloading
context ") will help?


Windows have filesystem audit. I believe Linux will have something
equivalent.

Or if there is another option please share with me.

Thanks for your help in advance.

-Utkarsh Dave


If you are interested to know what is filling Permgen...then:
First, look inside the Permgen and see what is growing over time. JMAP can
help you. Use jmap -permstat command to gather Permgen data - every 12hrs.
Then compare them. visualvm is another free and very nice option to look
inside Permgen.

You can also check heap dumps with Eclipse MAT and look for duplicate
classes. In your case, I suspect there will be multiple classloaders and
some of them will be inactive. Find out what is preventing those inactive
classloaders from garbage collected. You can use MAT's "shortest path to gc
roots" option to do that. They are the problem.

yourkit can also help you to find that out easily...jprofiler fans, please
don't get offended. ;)

Thanks!
Suvendu

Re: Tomcat is stopping on its own even though stop script is not executed

[Suvendu Sekhar Mondal]

Chaitanya,

On Thu, Jul 20, 2017 at 8:33 PM, Chaitanya Sabbineni
 wrote:
> Hi Chris,
>
> Stop script in the sense it's Catalina script only but we usually stop
> tomcat using the command Catalina.sh stop. But in our case we are not
> manually executing this script to stop tomcat and tomcat is stopping on its
> own.
>
> our main problem here us tomcat is stopping on its own and it needs a
> restart.
>
> If I understand you correct you are telling TimerThread that does not stop
> when the application is shut down. Can you let me know what actually the
> timer thread mean. And moreover if the timer thread didn't stop ideally
> tomcat shouldn't stop but in our case it's stopping.
>
> Yes my question is why Tomcat is being shut down at all.
>
> Yes when ever tomcat is stopping on own(not daily) it stops at 02:00 . You
> mentioned that your  guess is that we are using a service runner that is
> configured to bounce your services at 02:00.Can let me know what this
> service runner is and how to check it.
>

First, is that a full thread dump? One interesting thing in your dumps
is that there is no "main" thread. This is the first time I am seeing
a thread dump like this.

Second thing is, it seems that you are running some type of scheduler.
Can you please check if that is not doing anything crazy?
"PASS_DEFAULT_GROUP_com.j.passj.core.config.scheduler.LicenseRemovalAfterExpiry"
#53 daemon prio=5 os_prio=0 tid=0x7f882461a000 nid=0x1942 in
Object.wait() [0x7f8810b1f000]

One last thing, can you please check Tomcat is not shutting down due
to OutOfMemory?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Unable to install Tomcat 9 on Windows 10

[Suvendu Sekhar Mondal]

On Thu, Jul 20, 2017 at 7:53 PM, Mark Thomas  wrote:
> On 20/07/17 14:45, Roparzh Hemon wrote:
>>  Hello all,
>>
>>  I am currently unable to install Tomcat 9.0 on my Windows 10
>> system (I didn't install any other version of Tomcat so far).
>>  I've retried several times and the same problem appears over and
>> over again :
>>  The install process goes on smoothly with the install wizard, up
>> to the point where I see the following output :
>>
>>  Installing Tomcat9 service
>>  Apache Tomcat Setup
>>  Failed to install Tomcate9 service.
>>  Check your settings and permissions.
>>  Ignore and continue anyway (not recommended) ?
>>  Abort / Retry / Ignore
>>
>>How exactly should I "check my settings and permissions"  ?
>>Any help appreciated.
>
> You probably need to install with admin privs.
>
> Mark

Agreed.

Roparzh,
Sounds like you have downloaded the binary version of Tomcat. Can you
please right-click on it and check the properties? If you see a
"unblocked" checkbox, then unblock it first. Then try to install it
with admin privilege.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat 8.5.16 silent exit without any error logs on ubuntu 16.04.2

[Suvendu Sekhar Mondal]

>> >
>> > On Fri, Jul 7, 2017 at 12:40 AM, 李员外 <281170...@qq.com> wrote:
>> >
>> > > Hi, guys
>> > >
>> > >
>> > > I use tomcat 8.5.16, and I found at least 5 times tomcat silent exit
>> > > without any error logs after keep running 2-3 days, that means
>> > catalina.out
>> > > has no error info located in /opt/apache-tomcat-8.5.16/logs/.
>> > > And application log file also has no error info.
>> > > I do not know why? How should I to find out reason?
>> > >
>> > >
>> > > Please help me!
>> > > Thanks in advance!
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > > 做自己
>> >

Hi,

What is the JRE version? I hope you are not using
ExitOnOutOfMemoryError option [1]. If this is not the case, then my
hunch is some external process is killing the Tomcat and you need to
find it out. In Windows, whenever something like that happens, one
eventlog gets created. I am pretty sure there will be similar thing in
Ubuntu. I just Googled and saw this [2].

Another thing I would suggest, please monitor the memory usage of the
Tomcat process using some tool. Try to identify if it is getting
crashed after reaching to a certain threshold.

Thanks!
Suvendu

[1]http://www.oracle.com/technetwork/java/javase/8u92-relnotes-2949471.html
[2]http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: MaxThread Configurations Issue

[Suvendu Sekhar Mondal]

Shailesh,

On Tue, May 16, 2017 at 12:37 AM, Mark Eggers
 wrote:
> Shailesh,
>
> On 5/15/2017 8:22 AM, Shailesh Jain wrote:
>> We have updated the maxThread configuration to 10 at below place in
>> server.xml of DEV environment. However I was able to see more than 10
>> threads. I have also attached the server.xml
>>
>> I need your help to understand if I am doing something wrong or it is an
>> expected behaviour?
>>
>>
>> > redirectPort="8443" maxThreads="10"/>
>> 
>>
>
> Tomcat uses other threads besides the connector threads. This
> configuration only limits the number of connector threads.

Agreed. You can take a thread dumps and you will get to see all thread
details and probably what they are doing.

>
> . . . just my two cents
> /mde/
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tuning to accommodate Apache 2.4 event mpm

[Suvendu Sekhar Mondal]

John,

On Fri, Apr 28, 2017 at 11:48 AM, Rainer Jung  wrote:
> Am 27.04.2017 um 22:21 schrieb Mark Thomas:
>>
>> On 27/04/17 20:08, John Cartwright - NOAA Federal wrote:
>>>
>>> Thanks for your reply Mark!  My sysadmin tells me just that we're
>>> using "the defaults" for event_mpm.  However we are still using the
>>> BIO AJP connector:
>>>
>>> >> maxThreads="300" minSpareThreads="25"
>>> maxSpareThreads="75" connectionTimeout="2"/>
>>>
>>> Is there a way I can tell from the Tomcat side when there is thread
>>> starvation?
>>
>>
>> You'd need to look at the number of active connections. If it gets to
>> 300 then you likely have a problem.
>>
>> netstat might be the least invasive way of doing that. Other options
>> include the Manager app, JMX and thread dumps.
>>
>> Switching to NIO will allow you to server more connections with the same
>> number of threads.
>
>
> I assume you are using the latest mod_jk version?
>
> The current version of mod_jk also counts the total connection number (sum
> of connections from all Apache child processes to your Tomcat) and makes it
> available via the status worker for monitoring. The counting was a bit buggy
> in older versions, but should be correct in the latest one.
>
> Event MPM and mod_jk: the relevant phase in Apache is the handler phase,
> where the actual request forwarding is done by mod_jk. Things do not really
> need to change in the handler phase for proxying modules with event MPM.
> mod_jk does not need to know about the MPM or behave specific, the
> forwarding runs synchronously. By default mod_jk uses as many connections
> per Apache process as you have worker threads per Apache process configured.
> On the Tomcat side with BIO you need as many threads for the AJP connector
> as all you Apache fronting that Tomcat might have threads in total. Using
> the NIO connector on the Tomcat side is indeed better than BIO especially
> when you expect lots of concurrency (many Apache threads configured). Make
> sure you also use a recent Tomcat patch level then.
>
> What messages did you get in your mod_jk error log?

I also have the same question. Please check mod_jk log entries in the
time frame when you are getting 503s. It must have some clues.

We had similar type of problem a year ago(although we were using
mod_proxy_http). Later we found that backend Tomcat servers were not
responding to httpd's ping. As a result httpd was throwing 503s
sporadically. In our case long GC pauses were causing the problem. We
tuned GC and problem went away completely. You can check if you are
having something like that or not. Again you need to correlate 503
occurrences with GC logs.

https://tomcat.apache.org/connectors-doc/reference/workers.html

>
> For the mod_jk config, a good starter is the config files we bundle with the
> source download of mod_jk. You can also find them under
>
> http://svn.apache.org/viewvc/tomcat/jk/trunk/conf/
>
> These files set timeouts etc.
>
> Regards,
>
> Rainer
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: any way to get 20k request/seconds in tomcat

[Suvendu Sekhar Mondal]

Laxmi,

On Tue, Apr 25, 2017 at 7:20 AM, Laxmi Narayan  wrote:
> Hi ,
>
> I am looking to get some 20k or above response in tomcat.
>
> I am not able to find out best combination for that.
>
> My servlet will return just "hello-world" response.
>
>
> Keep learning keep moving .

It is hard to answer your question as you did not mentioned any
details about your current Tomcat config and what throughput you are
getting.

You mentioned that your servlet will be very simple, that means it
really comes down to the concurrency part. For a starter, you can try
to bump up the thread pool size. Still I do not think there is a
single switch by turning on which you can get the high throughput.
IMO, it can only be achieved through repeated "measure, change,
measure" approach. Because it is not only Tomcat configuration, you
need to take other things like heap sizing, GC policy, OS, Network and
hardware settings into consideration.

You can go through:
https://tomcat.apache.org/tomcat-8.0-doc/config/  

Re: Running multiple instance of tomcat on windows

[Suvendu Sekhar Mondal]

On Thu, Apr 20, 2017 at 7:19 PM, JUGAL SHAH  wrote:
> my first instance port are:
>
> 
>
>  connectionTimeout="2"
>redirectPort="8443" />
>
>  
>
>
> my Second  instance port are:
>
> 
>
> connectionTimeout="2"
>redirectPort="8444" />
>
> 
>
>
>
>
> when i run the second instance it says page can't be reached
> Their are many log files which one should I look into?
> and I am running tomcat version 9.0.0.M6
> I am running it directly on browser (localhost:8085)
>
> 8085 runs fine its the port mentioned in the original file
>
So you have two URLs now, assuming you are using HTTP:
localhost:8085 

Re: Running multiple instance of tomcat on windows

[Suvendu Sekhar Mondal]

Jugal,

On Thu, Apr 20, 2017 at 6:41 PM, JUGAL SHAH  wrote:
> I have to run multiple instance of tomcat server on a single system with
> window 8.1 OS.I searched the topic I got the solution to make 2 copies of
> tomcat and change the ports in the server.xml file I tried it but  it
> didn't work tomcat still continues to run only on port 8080.Any help?

Which ports(connector/shutdown/redirect) you have changed? Can you
please post server.xml for both of your Tomcat instances? Also please
check what Andre had suggested.

~Suvendu

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Ways to identify poorly designed client aplications sending request to Tomcat !

[Suvendu Sekhar Mondal]

>Memory heap dump generated is of
>Size: 787.3 MB Classes: 139k Objects: 19.3m Class Loader: 1.6k

>Overview shows 580.9 MB occupied by remainder's.

>Problem suspect:-
>465 MB occupied by remainder

Remainder section has retained a good chunk of memory. That indicates
lots of small objects are being created by different apps. Your "Live
Set" is not very big. What is the heap size? You also mentioned,
Tomcat process was consuming high CPU. If you have small heap and all
of it is filled up by live objects then JVM will run frequent GC to
clean up some space. In that case CPU usage will be high for Tomcat
process.

As Olaf indicated, you can try to increase heap size and see if the
problem goes away. But before that, I am curious to see what heap and
GC settings you are using. Please post that info.

Thanks!
Suvendu

On Thu, Mar 30, 2017 at 2:01 PM, Olaf Kock  wrote:
> Am 30.03.2017 um 01:33 schrieb Utkarsh Dave:
>> Hello all,
>>
>> My tomcat (7.0.72) hosts several web aplications in the server (based in
>> linux 6.8).
> [...]
>> Memory heap dump generated is of
>> Size: 787.3 MB Classes: 139k Objects: 19.3m Class Loader: 1.6k
> The combination of "hosts several web applications" and a heap space of
> this size does not convince me of a leak - it might be the memory
> requirement of one of the webapps. A leak is typically something that
> grows uncontrolled until you run out of heapspace, no matter how much
> you grow the available space.
>> In the thread dumps I see these threads repeatedly. I wonder these pointing
>> to com.rsa.sslj.x.
> You seem to be handling https requests from Tomcat. If you're not happy
> with the implementation of this endpoint/protocol you should move this
> to an Apache httpd or similar and just forward to tomcat, so that tomcat
> does not deal with encryption.
>
> As a conclusion: Your problem might not be poorly designed clients, it
> might be poorly equipped servers - I'd try to double the memory
> allocated to Tomcat's heap and potentially tune the garbage collector.
> If you run into problems, you might also identify one of the
> webapplications that eats up your resources (no matter what the clients
> do).
>
> Olaf
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Not Responding (Resolved)

[Suvendu Sekhar Mondal]

Well, I took thread dumps on couple of Tomcat JVMs(which were not
busy...JVMs on my system) and for all of them it showed thread opened
socket and accepting connection(StandardServer.java:446):

at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0xfbf67198> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)

But I don't see that for problematic JVM. It was stuck before
that(StandardServer.java:427) step.

at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)

That's why I suspect JVM was not properly up and running. In fact, I
never saw a main thread in TIMED_WAITING waiting state. Again, I might
be wrong. :)

Thanks!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Not Responding (Resolved)

[Suvendu Sekhar Mondal]

Sorry for late "chime in".

It seems that your Tomcat JVM was not up properly. From both thread
dumps showing main thread in TIMED_WAITING waiting state not in
runnable state :

"main" #1 prio=5 os_prio=0 tid=0x01198000 nid=0x2340 waiting
on condition [0x0132e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
at org.apache.catalina.startup.Catalina.await(Catalina.java:743)
at org.apache.catalina.startup.Catalina.start(Catalina.java:689)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:355)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:495)

   Locked ownable synchronizers:
- None

I went by the source code(which is little bit outdated):
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.tomcat/tomcat-catalina/8.0.24/org/apache/catalina/core/StandardServer.java/

Did you check stderr/stdout log? It should log some error.

Here is snippet of main thread from running Tomcat JVM:

"main" #1 prio=5 os_prio=0 tid=0x01084000 nid=0x2008 runnable
[0x0107e000]
   java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0xfbf67198> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
at org.apache.catalina.startup.Catalina.await(Catalina.java:713)
at org.apache.catalina.startup.Catalina.start(Catalina.java:659)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)

   Locked ownable synchronizers:
- None

Thanks!
Suvendu

On Sat, Mar 25, 2017 at 2:51 AM, Igal @ Lucee.org  wrote:
> Chris,
>
> On 3/24/2017 2:13 PM, Christopher Schultz wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> Igal,
>>
>> On 3/24/17 1:22 PM, Igal @ Lucee.org wrote:
>>>
>>> I've traced the issue to an NPE thrown from my servlet.  I patched
>>> it (https://github.com/lucee/Lucee/commit/0f30a7ef) and now it
>>> works fine.
>>
>> That's good to know, but an NPE thrown from an application shouldn't
>> be able to lock-up Tomcat. Can you reproduce this using an SSCCE[1]?
>
>
> I agree.  That's the reason I created a simple html file and tested it, as
> those do not go through the Lucee servlet.
>
> I will try to produce a reduced test case, but not sure how simple that
> would be.
>
>
> Igal
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JMX currentThreadsBusy less than connections/requests when use APR connector

[Suvendu Sekhar Mondal]

Linbo,

"currentThreadsBusy" is number of busy threads. These are the threads
are being actively use. If you are seeing this count > 0 for long
time(depending on your application type), then most likely you have
"hung thread". In that case thread dump analysis will show you root of
the problem.

"currentThreadCount" is current threads in thread pool. To track
thread pool usage, you can use this counter.

Thanks!
Suvendu

On Wed, Mar 8, 2017 at 8:44 AM, linbo liao  wrote:
> Hi,
>
> I setup local environment to test Tomcat monitor.
>
> The Environment:
>
> Tomcat: 8.5.5
> VM: Ubuntu 14.04.1 LTS
> HTTP PORT: 8080
> IP: 10.211.55.4
>
> Tomcat use APR connector, I test the tomcat via ab command, find JMX
> currentThreadsBusy < 10 all of the time.
>
> ab -n 10 -c 100 10.211.55.4:8080/
>>
>
> I tried to search the reason but without the result. For BIO each thread to
> handle one connection, so currentThreadsBusy can show the performance of
> tomcat.
>
> But for APR connector, what's the meaning of currentThreadsBusy?
>
> Thanks in advance.
>
> Thanks,
> Linbo

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to enable Native Memory Tracking(NMT) in Tomcat?

[Suvendu Sekhar Mondal]

It worked, Mark! When I launched Tomcat from command line, it fired up
Bootstrap loader. Then JMC can print NMT statistics.

I believe tracking NMT is not a common need. Still, it would be
awesome if we can do the same thing with changes in Apache Commons
Daemon.

Thanks!

On Fri, Mar 3, 2017 at 2:06 PM, Mark Thomas <ma...@apache.org> wrote:
> On 03/03/17 06:20, Suvendu Sekhar Mondal wrote:
>> Mark,
>> I am running Tomcat as a Windows service.
>
> Then I suspect that supporting this would require changes in Apache
> Commons Daemon which is what Tomcat uses to start the Windows service.
>
> It would probably be quicker for you to run Tomcat from the command line
> to get the debugging information you require - assuming you have a
> one-off need. If you want ongoing monitoring then Daemon changes is
> would be the better option.
>
> Mark
>
>>
>> Thanks!
>> Suvendu
>>
>> On Thu, Mar 2, 2017 at 8:08 PM, Mark Thomas <ma...@apache.org> wrote:
>>> On 02/03/17 10:54, Suvendu Sekhar Mondal wrote:
>>>> Hello Everyone,
>>>>
>>>> I am new here. :)
>>>>
>>>> Environment:
>>>> Java Version: Java HotSpot(TM) 64-Bit Server VM version 25.91-b15
>>>> (Java version 1.8.0_91-b15)
>>>> Tomcat Version: Tomcat 8.0.20
>>>> OS Version: Microsoft Windows 8.1 Enterprise
>>>>
>>>> I am trying to enable Native Memory Tracking(NMT) to get internal
>>>> memory usage deatils about Tomcat process which is running on my
>>>> system. I have added following flags to Tomcat's Java options:
>>>> -XX:+UnlockDiagnosticVMOptions
>>>> -XX:NativeMemoryTracking=summary
>>>> -XX:NativeMemoryTracking=detail
>>>>
>>>> After that I restarted Tomcat. When I tried to use "VM.native_memory"
>>>> command either from JCMD or JMC, I am getting "Native memory tracking
>>>> is not enabled" message. When I shutdown Tomcat, following message is
>>>> getting printed on STDERR log:
>>>> "Java HotSpot(TM) 64-Bit Server VM warning: Native Memory Tracking did
>>>> not setup properly, using wrong launcher?".
>>>>
>>>> I found this article which discussed about NMT problem in custom JVM 
>>>> launcher:
>>>> https://blogs.oracle.com/poonam/entry/using_nmt_with_custom_jvm
>>>>
>>>> My question is, does that stands true for Tomcat also? Is there any
>>>> other way to enable NMT in Tomcat?
>>>
>>> No idea. How did you start Tomcat?
>>>
>>> Mark
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org