Hi,
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Thursday, October 10, 2019 3:54 AM
To: users@tomcat.apache.org
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
On 09/10/2019 22:58, Rhuberg,Anthony wrote:
> StandardRoot.gc() unconditionally closes the web application jars in Tomcat
> 9... every 10s by default or configurable by changing
> backgroundProcessorDelay.
>
> So then the problem is.... Our web application is calling
> class.getResourceAsStream() to read some static files... and this is causing
> the jars to be read again?
It sounds like it.
> This was not a problem with Tomcat 7 because the jars were not closed if they
> were read within the jarOpenInterval setting?
Or that the frequency that the JARs were closed was low enough that the
performance impact was not noticed.
Mark
>
> Thanks,
> Tony
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 5:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 21:03, Rhuberg,Anthony wrote:
>> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
>> <Context docBase="${STP_HOME}/app/webapps/Clinicals"
>> swallowOutput="true" backgroundProcessorDelay="90">
>>
>> Not sure if this is the context reload trigger... i.e. the
>> webappLoader.backgroundProcess method is triggered every 90 seconds...
>
> It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
>
> Mark
>
> ---------------------------------------------------------------------
Here is an update as to the causes of the repeated reloads our web application
jars. Some of the following may be "bad" practices (at least not optimal), but
seem to be commonplace. In other cases, the behavior was unexpected and may
warrant a change to WebResourceRoot to allow the jars to remain open or
otherwise control that behavior.
Have 2 questions:
1. Could the WebResourceRoot be changed to provide a way to control when the
jars are closed?
2. Any concerns with any of the solutions below? Has anyone used these in the
past or currently and found any problems?
1. JAXP related factories
We frequently use DocumentBuilderFactory.newInstance() to obtain an instance of
the factory implementation and others like TransformerFactory, XPathFactory,
DatatypeFactory, and SAXParserFactory. This will trigger a jar reload when the
service loader attempts to find the implementation class. We decided to set the
applicable system properties to avoid the search and thus eliminate this reason
for the jar reload. See below for the factories configured; this change reduced
JVM CPU usage percentage in a load test by 26% (using Tomcat 7, so expect
similar in Tomcat 9, plus the elimination of the jar reload).
2. SOAP web service endpoints
Each time we invoke a SOAP web service we are creating an instance of
javax.xml.ws.Service and a client proxy that implements the applicable web
service contract (Service.getPort). When these objects are created the service
loader is used to find the implementation for the web service client. We are
using the Apache CXF client
(cxf-rt-frontend-jaxws/META-INF/services/javax.xml.ws.spi.Provider). In
addition, the JAXB bindings are created. These actions occur every time we
invoke a SOAP web service; this subsequently searches for the implementation
classes triggering the web application jars to be reloaded. We decided to
create the service end-points once and reuse. Creating the objects once and
reusing avoids the cost of repeatedly creating the instances. The CXF instances
are thread safe
(https://cxf.apache.org/faq.html#FAQ-AreJAX-WSclientproxiesthreadsafe?).
Testing to verify the objects are thread safe is in progress.
3. When we invoke a REST web service, we create an instance of the Spring
RestTemplate. The template then registers various mappers; the
MappingJackson2HttpMessageConverter uses
Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable() to register a
few optional modules. Each time the template is created a classpath search is
performed that can trigger the jars to be reloaded. See:
https://github.com/spring-projects/spring-framework/search?utf8=%E2%9C%93&q=registerWellKnownModulesIfAvailable&type=.
The optional modules include: Jdk7Module, Jdk8Module, and JavaModule. Adding
these jars allows the classes to be found and then cached, thus preventing
subsequent jar reloads: jackson-datatype-jsr310-2.10.1.jar
jackson-datatype-jdk7-2.6.7.jar, and jackson-datatype-jdk8-2.10.1.jar.
4. When transforming XML documents, we repeatedly read the .xsl file using
getResourceAsStream(sometransform.xsl). This is cached by default for 5
seconds. We increased the associated Resources cacheTtl to 60 minutes. Not sure
if this will be our solution.
5. We implemented an override to StandardRoot to simply no-op the gc()
operation; then we configure the related Resources to use that class. We have
not decided to use this; it helps set the performance baseline back to the
behavior we saw in Tomcat 7.
Related to #1 above...
JAXP properties
javax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
javax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
javax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
javax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
javax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl
javax.xml.stream.XMLOutputFactory=com.sun.xml.internal.stream.XMLOutputFactoryImpl
javax.xml.stream.XMLEventFactory=com.sun.xml.internal.stream.events.XMLEventFactoryImpl
javax.xml.xpath.XPathFactory\:http\://java.sun.com/jaxp/xpath/dom=com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl
javax.xml.validation.SchemaFactory\:http\://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory
org.apache.xml.dtm.DTMManager=org.apache.xml.dtm.ref.DTMManagerDefault
CONFIDENTIALITY NOTICE This message and any included attachments are from
Cerner Corporation and are intended only for the addressee. The information
contained in this message is confidential and may constitute inside or
non-public information under international, federal, or state securities laws.
Unauthorized forwarding, printing, copying, distribution, or use of such
information is strictly prohibited and may be unlawful. If you are not the
addressee, please promptly delete this message and notify the sender of the
delivery error by e-mail or you may call Cerner's corporate offices in Kansas
City, Missouri, U.S.A at (+1) (816)221-1024.
