Jaxrs/Jaxb unmarshaller code regularly causing performance problems & thread
lock-up
------------------------------------------------------------------------------------
Key: CXF-3180
URL: https://issues.apache.org/jira/browse/CXF-3180
Project: CXF
Issue Type: Bug
Affects Versions: 2.3.1, 2.2.12, 2.2.11, 2.3.0, 2.2.10, 2.2.9, 2.2.8,
2.2.7, 2.2.6
Reporter: Willem Jiang
Assignee: Willem Jiang
Fix For: 2.3.2, 2.4
Please find below the details of the XML parsing performance bottleneck we are
seeing when processing REST POST requests with XML bodies.
The performance issue is caused by the fact that CXF code creates a new JAXB
Unmarshaller for every incoming request. The Unmarshaller is created quickly
enough but when it is used to do the unmarshalling, it needs to create a Xerces
SAXParser, which in turn calls DTDDVFactory.getInstance() and that method is
synchronized. When we try to process more than 200-300 requests per second we
start seeing a lot of threads blocked in DTDDVFactory.getInstance().
The thread stack of one of the blocked threads is:
1390532...@qtp-69728034-390" prio=10 tid=0x00002aab449c0000 nid=0x28f4 waiting
for monitor entry [0x000000004da26000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)
waiting to lock <0x00002aaab279ed48> (a java.lang.Class for
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
at
com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
at
com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at
com.sun.org.apache.xerces.internal.parsers.ObjectFactory.newInstance(ObjectFactory.java:349)
at
com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:154)
at
com.sun.org.apache.xerces.internal.parsers.ObjectFactory.createObject(ObjectFactory.java:97)
at
com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:102)
at
com.sun.org.apache.xerces.internal.parsers.SAXParser.<init>(SAXParser.java:87)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.<init>(SAXParserImpl.java:332)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.<init>(SAXParserImpl.java:122)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:76)
at
javax.xml.bind.helpers.AbstractUnmarshallerImpl.getXMLReader(AbstractUnmarshallerImpl.java:75)
at
javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:184)
at
javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:191)
at
org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:216)
at
org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:186)
at
org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:157)
at
org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:947)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:560)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:525)
at
org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:230)
at
org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:88)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243)
locked <0x00002aab1e41a0e8> (a org.apache.cxf.phase.PhaseInterceptorChain)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:109)
at
org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:312)
...
The application where the problem is seen defines an XSD file and uses theJAXB
compiler to create the data binding java class. It then creates a "RESTResouce"
java class that is CXF/JAXRS annotated and contains a method that defines the
processing of incoming POST requests. That method has one argument, the
JAXB-compiled class, which tells CXF to unmarshall the XML body of the incoming
request.
Just in case, you need more context, here is the top portion of the stack of
the thread that is inside the synchronized method:
"1840040...@qtp-69728034-391" prio=10 tid=0x00002aab44810000 nid=0x2dfa waiting
on condition [0x000000005be59000]
java.lang.Thread.State: RUNNABLE
at
org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:211)
at
org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:381)
at
org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:457)
at
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:410)
at
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:398)
at
org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:105)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.eclipse.osgi.internal.loader.BundleLoader.loadClass(BundleLoader.java:326)
at
org.eclipse.osgi.framework.internal.core.BundleHost.loadClass(BundleHost.java:231)
at
org.eclipse.osgi.framework.internal.core.AbstractBundle.loadClass(AbstractBundle.java:1193)
at
org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)
at
org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.findProviderClass(ObjectFactory.java:395)
at
com.sun.org.apache.xerces.internal.impl.dv.ObjectFactory.newInstance(ObjectFactory.java:350)
at
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:59)
locked <0x00002aaab279ed48> (a java.lang.Class for
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory.getInstance(DTDDVFactory.java:44)
locked <0x00002aaab279ed48> (a java.lang.Class for
com.sun.org.apache.xerces.internal.impl.dv.DTDDVFactory)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.<init>(XML11Configuration.java:538)
at
com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:125)
at
com.sun.org.apache.xerces.internal.parsers.XIncludeAwareParserConfiguration.<init>(XIncludeAwareParserConfiguration.java:86)
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
