This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
<Context docBase="${STP_HOME}/app/webapps/Clinicals" swallowOutput="true"
backgroundProcessorDelay="90">
Not sure if this is the context reload trigger... i.e. the
webappLoader.backgroundProcess method is triggered every 90 seconds... the
reloadable property on Context is NOT set (default = false). So then how does
modified() evaluate to true?
The
https://github.com/apache/tomcat/blob/65aec2ac7fb4980694291e66efd360195dfa8d73/java/org/apache/catalina/loader/WebappLoader.java...
/**
* Execute a periodic task, such as reloading, etc. This method will be
* invoked inside the classloading context of this container. Unexpected
* throwables will be caught and logged.
*/
@Override
public void backgroundProcess() {
if (reloadable && modified()) {
try {
Thread.currentThread().setContextClassLoader
(WebappLoader.class.getClassLoader());
if (context != null) {
context.reload();
}
} finally {
if (context != null && context.getLoader() != null) {
Thread.currentThread().setContextClassLoader
(context.getLoader().getClassLoader());
}
}
}
}
I do not understand how modified = true. Why would the jar date/time be changed?
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/loader/WebappClassLoaderBase.java
/**
* Have one or more classes or resources been modified so that a reload
* is appropriate?
* @return <code>true</code> if there's been a modification
*/
public boolean modified() {
if (log.isDebugEnabled())
log.debug("modified()");
for (Entry<String,ResourceEntry> entry : resourceEntries.entrySet()) {
long cachedLastModified = entry.getValue().lastModified;
long lastModified = resources.getClassLoaderResource(
entry.getKey()).getLastModified();
if (lastModified != cachedLastModified) {
if( log.isDebugEnabled() )
log.debug(sm.getString("webappClassLoader.resourceModified",
entry.getKey(),
new Date(cachedLastModified),
new Date(lastModified)));
return true;
}
}
// Check if JARs have been added or removed
WebResource[] jars = resources.listResources("/WEB-INF/lib");
// Filter out non-JAR resources
int jarCount = 0;
for (WebResource jar : jars) {
if (jar.getName().endsWith(".jar") && jar.isFile() &&
jar.canRead()) {
jarCount++;
Long recordedLastModified =
jarModificationTimes.get(jar.getName());
if (recordedLastModified == null) {
// Jar has been added
log.info(sm.getString("webappClassLoader.jarsAdded",
resources.getContext().getName()));
return true;
}
if (recordedLastModified.longValue() != jar.getLastModified()) {
// Jar has been changed
log.info(sm.getString("webappClassLoader.jarsModified",
resources.getContext().getName()));
return true;
}
}
}
if (jarCount < jarModificationTimes.size()){
log.info(sm.getString("webappClassLoader.jarsRemoved",
resources.getContext().getName()));
return true;
}
// No classes have been modified
return false;
}
-----Original Message-----
From: Rhuberg,Anthony
Sent: Wednesday, October 09, 2019 3:08 PM
To: Tomcat Users List <[email protected]>
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
On the other thread: Is this genuine class loading (in which case the response
below is correct) or is the application reading the .class files as if they
were resources (in which case a different answer applies)?
Mostly, we are talking about the default class loader; however, the application
does read resources from the jar files - mostly static files that are stored
within the package structure are read (for example: log4j, property files,
other static content). Would this affect the reload of the jar files?
-----Original Message-----
From: Rhuberg,Anthony
Sent: Wednesday, October 09, 2019 2:53 PM
To: Tomcat Users List <[email protected]>
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Just noticed another thread on this topic: RE: Tomcat discards and reloads the
jar files from the webapps folder.
Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the
open/close cycle of the jars.
The other thread also mentioned a potential configuration to keep the jars open
longer by default. This could be helpful - not sure we need to close them at
all.
Alternatively, would there be any gain to 'explode' the jars into class files
on the filesystem?
-----Original Message-----
From: Rhuberg,Anthony <[email protected]>
Sent: Wednesday, October 09, 2019 2:17 PM
To: [email protected]
Subject: Performance test with Tomcat 9 shows increased cpu/disk usage because
of repeated opening/closing of jars in WEB-INF/lib
Background:
In the last few months we migrated our web application from Tomcat 7.0.55 to
Tomcat 9.0.19 (26). That transition was relatively straightforward until we
reviewed the results of our performance tests. Those tests showed an increase
in CPU usage and disk I/O on our Windows 2012 server. When we switch back to
Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to
previous tests).
As we investigated the cause of the increase in cpu/disk usage we found the
tomcat process was repeatedly opening and closing the jars of our web
application. The process profile showed all the jars in WEB-INF/lib being
closed and reopened within seconds while the application was processing
requests (and to some extent when quite). Now we are trying to determine why.
We have a general understanding that the WebappClassLoaderBase.java
implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for
the configuration of jarOpenInterval (which we used the default = 90s). The
latest implementation appears to track the jar modification times with a hash
map called jarModificationTimes, but we are unable to find a similar
time-to-live configuration or what triggers the lifecycle listener to close the
jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we
understand this lifecycle behavior.
The server is configured mostly with default settings. The Context reloadable
property is NOT set. We have tried increasing the cache time of the Resources
cacheTtl property as the default of 5s seemed close to the time the jars were
reloaded. We have not been successful.
Question:
What is the cause of the repeated reloading of our web application jars? What
can trigger the class loader to close all the file handles related to the jars
in the web application WEB-INF/lib directory - and then open them again within
a few seconds?
CONFIDENTIALITY NOTICE This message and any included attachments are from
Cerner Corporation and are intended only for the addressee. The information
contained in this message is confidential and may constitute inside or
non-public information under international, federal, or state securities laws.
Unauthorized forwarding, printing, copying, distribution, or use of such
information is strictly prohibited and may be unlawful. If you are not the
addressee, please promptly delete this message and notify the sender of the
delivery error by e-mail or you may call Cerner's corporate offices in Kansas
City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
