[
https://issues.apache.org/jira/browse/TAP5-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Douglas Hauge updated TAP5-1650:
--------------------------------
Attachment: tapestry-deadlock-fix-1.patch
Attached is a potential fix for the deadlock on startup (at least after
applying the patch, we are no longer able to reproduce the deadlock). The
problem was that there were multiple orders in which the 3 Tapestry classloader
locks could be acquired.
# standard: (prop expr conduit CL) -> (component CL) -> (service CL)
# problematic: (componet CL) -> (prop expr conduit CL)
The second happens when *ComponentInstantiatorSourceImpl.getInstantiator* is
called, but while assembling the component classes a service class needs to be
loaded. The rough path for this is
# ComponentInstantiatorSourceimpl.getInstantiator
# -> ComponentInstantiatorSourceImpl.createInstantiatorForClass
# -> PlasticManager.getClassInstantiator (component classloader manager, not
that in 'proxyFactory')
# -> PlasticClassPool.getClassInstantiator
# -> PlasticClassLoader.loadClass (acquires component class loader lock)
# -> ComponentInstantiatorSourceImpl.transform -> RegistryImpl.run -> ... ->
DefaultInjectionProvider.provideInjection
# -> ComponentClassCacheImpl.forName
# -> ComponentClassCacheImpl.lookupClassForType (gets the property expression
conduit class loader from 'PlasticProxyFactory')
# -> PlasticInternalUtils.toClass -> Class.forName (acquires the lock on the
passed in class loader, the property expression conduit one).
The exact stacks of the deadlocked threads vary a little, but look similar to:
Bad lock order:
{code}
"TP-Processor44":
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.tapestry5.internal.plastic.PlasticInternalUtils.toClass(PlasticInternalUtils.java:306)
at
org.apache.tapestry5.internal.services.ComponentClassCacheImpl.lookupClassForType(ComponentClassCacheImpl.java:78)
at
org.apache.tapestry5.internal.services.ComponentClassCacheImpl.forName(ComponentClassCacheImpl.java:65)
at $ComponentClassCache_1089cca7fe04a.forName(Unknown Source)
at
org.apache.tapestry5.internal.services.DefaultInjectionProvider.provideInjection(DefaultInjectionProvider.java:65)
at $InjectionProvider2_1089cca7fe169.provideInjection(Unknown Source)
at $InjectionProvider2_1089cca7fe15d.provideInjection(Unknown Source)
at
org.apache.tapestry5.internal.transform.InjectWorker$2.run(InjectWorker.java:73)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at
org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at
org.apache.tapestry5.internal.transform.InjectWorker.transform(InjectWorker.java:67)
at $ComponentClassTransformWorker2_1089cca7fe165.transform(Unknown Source)
at $ComponentClassTransformWorker2_1089cca7fe157.transform(Unknown Source)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$3.run(ComponentInstantiatorSourceImpl.java:320)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at
org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.transform(ComponentInstantiatorSourceImpl.java:273)
at
org.apache.tapestry5.internal.plastic.PlasticClassPool.loadAndTransformClass(PlasticClassPool.java:313)
at
org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:38)
- locked <0x00000003027edc98> (a
org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.tapestry5.internal.plastic.PlasticClassPool.getClassInstantiator(PlasticClassPool.java:416)
- locked <0x00000003027edc98> (a
org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at
org.apache.tapestry5.plastic.PlasticManager.getClassInstantiator(PlasticManager.java:189)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:228)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:222)
at
org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at
org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.invoke(PerThreadOperationTracker.java:87)
at org.apache.tapestry5.ioc.internal.RegistryImpl.invoke(RegistryImpl.java:1010)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.createInstantiatorForClass(ComponentInstantiatorSourceImpl.java:220)
at
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.getInstantiator(ComponentInstantiatorSourceImpl.java:210)
- locked <0x0000000304c07bc8> (a
org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl)
at $ComponentInstantiatorSource_1089cca7fe152.getInstantiator(Unknown Source)
{code}
Good lock order:
{code}
"TP-Processor100":
at
org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:31)
- waiting to lock <0x00000003027edc98> (a
org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x000000039f889ab8> (a
org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at
org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:46)
- locked <0x000000039f889ab8> (a
org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at $InternalPropertyConduit_1089cca7ffb3f.getRoot(Unknown Source)
at $InternalPropertyConduit_1089cca7ffb3f.get(Unknown Source)
at org.apache.tapestry5.internal.bindings.PropBinding.get(PropBinding.java:59)
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.readFromBinding(ParameterWorker.java:254)
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.get(ParameterWorker.java:372)
at lithium.tapestry.mixins.IsVisible.getfieldvalue_isVisible(IsVisible.java)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java:32)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.invokeComponent(ComponentPageElementImpl.java:174)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$AbstractPhase.invoke(ComponentPageElementImpl.java:143)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.render(ComponentPageElementImpl.java:181)
at
org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at
org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
{code}
> On a cold start with a large number of incoming requests, Tapestry can
> deadlock inside PlasticClassLoader/PlasticClassPool
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: TAP5-1650
> URL: https://issues.apache.org/jira/browse/TAP5-1650
> Project: Tapestry 5
> Issue Type: Bug
> Components: plastic
> Affects Versions: 5.3
> Reporter: Howard M. Lewis Ship
> Assignee: Howard M. Lewis Ship
> Priority: Critical
> Labels: deadlock, performance
> Attachments: tapestry-deadlock-fix-1.patch
>
>
> Discovered during load testing; the scenario is a cold start for the
> application then hitting it with large numbers of concurrent requests with no
> ramp up. The deadlocks involve two methods:
> PlasticClassLoader.loadClass()
> PlasticClassPool.getClassInstantiator()
> A proper solution would to make these methods share a single explicit Lock,
> rather than be synchronized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
