[
https://issues.apache.org/struts/browse/WW-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=44742#action_44742
]
Aner Perez commented on WW-2753:
--------------------------------
I just spent a couple of days tracking down what I think is exactly this same
issue (very similar stack traces and symptoms).
In our case it turned out to be a nasty side effect of having "struts.devMode"
set to "true" in struts,xml (or struts.properties).
The xwork code in its infinite wisdom decided that if you are in devMode, it
will not cache resourceBundles. In order to do this, it clears the resource
bundles cache each time a localized message is accessed. Not being content
with clearing their own cache, they also decided that if the webapp server you
are using is tomcat, they will also manually clear out tomcats
WebappClassLoader resource cache with no regards to how this may affect tomcat.
One of the ways this affects Tomcat is that there is a list of resource file
names and time stamps that are maintained in an array. Normally each item
would be located only once in this array but since the cache is being cleared
behind their backs, the code is actually adding the same file names and
modified dates over and over again. This makes the code run slower and slower
as time goes by and the localized text is accessed over and over.
The problematic code is in
com.opensymphony.xwork2.util.LocalizedTextUtil.reloadBundles() which is called
from several places. Here is a snippet from version 2.1.1:
708 private static void reloadBundles() {
709 if (reloadBundles) {
710 try {
711 clearMap(ResourceBundle.class, null, "cacheList");
712
713 // now, for the true and utter hack, if we're running
in tomcat, clear
714 // it's class loader resource cache as well.
715 clearTomcatCache();
716 }
717 catch (Exception e) {
718 LOG.error("Could not reload resource bundles", e);
719 }
720 }
721 }
722
723
724 private static void clearTomcatCache() {
725 ClassLoader loader =
Thread.currentThread().getContextClassLoader();
726 // no need for compilation here.
727 Class cl = loader.getClass();
728
729 try {
730 if
("org.apache.catalina.loader.WebappClassLoader".equals(cl.getName())) {
731 clearMap(cl, loader, "resourceEntries");
732 } else {
733 if (LOG.isDebugEnabled()) {
734 LOG.debug("class loader " + cl.getName() + " is not
tomcat loader.");
735 }
736 }
737 }
738 catch (Exception e) {
739 LOG.warn("couldn't clear tomcat cache", e);
740 }
741 }
The clearTomcatCache() method being called on line 715 should not exist! This
is not a problem xwork should be trying to "solve".
Hope this helps someone.
> Threads are blocked in LocalizedTextUtil.findResourceBundle leading to very
> bad response times
> ----------------------------------------------------------------------------------------------
>
> Key: WW-2753
> URL: https://issues.apache.org/struts/browse/WW-2753
> Project: Struts 2
> Issue Type: Bug
> Affects Versions: 2.0.11.1
> Environment: RHEL3 with apache2.0 and tomcat 6.0.13, with an average
> load of 30-40 req per sec. We are using struts 2.0.11.1, xwork2.0.4,tiles
> 2.0.6.freemarker2.3.13.
> Reporter: kirti teja
> Priority: Critical
>
> we have encountered a production incident of very slow response times over 5
> mins per page. Investagtion lead to understanding that a huge number of
> threads are getting blocked on a synchronised block in
> LocalizedTextUtil.findResourceBundle. Attached is the below stack trace. We
> have encoutnered this with two different products in our company.
> "TP-Processor678" daemon prio=10 tid=0x5c5aa1d8 nid=0x4959 waiting for
> monitor entry [0x52fac000..0x52fae6f0]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findResourceBundle(LocalizedTextUtil.java:218)
> - waiting to lock <0x67db2650> (a java.util.HashMap)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.getMessage(LocalizedTextUtil.java:602)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:643)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:689)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:360)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:293)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:173)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:88)
> at com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:71)
> at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:851)
> at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1253)
> at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
> at
> com.opensymphony.xwork2.util.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:81)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1305)
> at
> com.opensymphony.xwork2.util.CompoundRootAccessor.callMethod(CompoundRootAccessor.java:196)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.ASTMethod.getValueBody(ASTMethod.java:90)
> at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
> at ognl.SimpleNode.getValue(SimpleNode.java:258)
> at ognl.Ognl.getValue(Ognl.java:494)
> at com.opensymphony.xwork2.util.OgnlUtil.getValue(OgnlUtil.java:194)
> at
> com.opensymphony.xwork2.util.OgnlValueStack.findValue(OgnlValueStack.java:238)
> at org.apache.struts2.components.Property.start(Property.java:136)
> at
> org.apache.struts2.views.freemarker.tags.CallbackWriter.onStart(CallbackWriter.java:72)
> at freemarker.core.Environment.visit(Environment.java:294)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.visit(Environment.java:297)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.process(Environment.java:188)
> at freemarker.template.Template.process(Template.java:237)
> at
> org.apache.struts2.views.freemarker.FreemarkerResult.doExecute(FreemarkerResult.java:168)
> at
> org.apache.struts2.tiles.StrutsTilesRequestContext.include(StrutsTilesRequestContext.java:118)
> "TP-Processor677" daemon prio=10 tid=0x5c823708 nid=0x4958 waiting for
> monitor entry [0x538fd000..0x538ff670]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findResourceBundle(LocalizedTextUtil.java:218)
> - waiting to lock <0x67db2650> (a java.util.HashMap)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.getMessage(LocalizedTextUtil.java:602)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:393)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:293)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:173)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:88)
> at com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:71)
> at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:851)
> at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1253)
> at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
> at
> com.opensymphony.xwork2.util.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:81)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1305)
> at
> com.opensymphony.xwork2.util.CompoundRootAccessor.callMethod(CompoundRootAccessor.java:196)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.ASTMethod.getValueBody(ASTMethod.java:90)
> at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
> at ognl.SimpleNode.getValue(SimpleNode.java:258)
> at ognl.Ognl.getValue(Ognl.java:494)
> at com.opensymphony.xwork2.util.OgnlUtil.getValue(OgnlUtil.java:194)
> at
> com.opensymphony.xwork2.util.OgnlValueStack.findValue(OgnlValueStack.java:238)
> at org.apache.struts2.components.Property.start(Property.java:136)
> at
> org.apache.struts2.views.freemarker.tags.CallbackWriter.onStart(CallbackWriter.java:72)
> at freemarker.core.Environment.visit(Environment.java:294)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.visit(Environment.java:297)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.ConditionalBlock.accept(ConditionalBlock.java:79)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.process(Environment.java:188)
> at freemarker.template.Template.process(Template.java:237)
> at
> org.apache.struts2.components.template.FreemarkerTemplateEngine.renderTemplate(FreemarkerTemplateEngine.java:148)
> at org.apache.struts2.components.UIBean.mergeTemplate(UIBean.java:530)
> at org.apache.struts2.components.UIBean.end(UIBean.java:484)
> at
> org.apache.struts2.views.freemarker.tags.CallbackWriter.afterBody(CallbackWriter.java:83)
> at freemarker.core.Environment.visit(Environment.java:299)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.visit(Environment.java:297)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.process(Environment.java:188)
> at freemarker.template.Template.process(Template.java:237)
> at
> org.apache.struts2.views.freemarker.FreemarkerResult.doExecute(FreemarkerResult.java:168)
> "TP-Processor572" daemon prio=10 tid=0x5e507a48 nid=0x3730 waiting for
> monitor entry [0x563fd000..0x563ff5f0]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findResourceBundle(LocalizedTextUtil.java:218)
> - locked <0x67db2650> (a java.util.HashMap)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.getMessage(LocalizedTextUtil.java:602)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:661)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:689)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:689)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:689)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findMessage(LocalizedTextUtil.java:689)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:360)
> at
> com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:293)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:173)
> at
> com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:88)
> at com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:71)
> at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:851)
> at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1253)
> at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
> at
> com.opensymphony.xwork2.util.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:81)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1305)
> at
> com.opensymphony.xwork2.util.CompoundRootAccessor.callMethod(CompoundRootAccessor.java:196)
> at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1329)
> at ognl.ASTMethod.getValueBody(ASTMethod.java:90)
> at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
> at ognl.SimpleNode.getValue(SimpleNode.java:258)
> at ognl.Ognl.getValue(Ognl.java:494)
> at com.opensymphony.xwork2.util.OgnlUtil.getValue(OgnlUtil.java:194)
> at
> com.opensymphony.xwork2.util.OgnlValueStack.findValue(OgnlValueStack.java:238)
> at org.apache.struts2.components.Property.start(Property.java:136)
> at
> org.apache.struts2.views.freemarker.tags.CallbackWriter.onStart(CallbackWriter.java:72)
> at freemarker.core.Environment.visit(Environment.java:294)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.visit(Environment.java:297)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.process(Environment.java:188)
> at freemarker.template.Template.process(Template.java:237)
> at
> org.apache.struts2.views.freemarker.FreemarkerResult.doExecute(FreemarkerResult.java:168)
> at
> org.apache.struts2.tiles.StrutsTilesRequestContext.include(StrutsTilesRequestContext.java:118)
> at
> org.apache.struts2.tiles.StrutsTilesRequestContext.dispatch(StrutsTilesRequestContext.java:84)
> at
> org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:457)
> at
> org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:140)
> at
> org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:117)
> at
> org.apache.tiles.jsp.taglib.RenderTagSupport.execute(RenderTagSupport.java:171)
> at
> org.apache.tiles.jsp.taglib.RoleSecurityTagSupport.doEndTag(RoleSecurityTagSupport.java:75)
> at
> org.apache.tiles.jsp.taglib.ContainerTagSupport.doEndTag(ContainerTagSupport.java:80)
> at
> freemarker.ext.jsp.TagTransformModel$TagWriter.endEvaluation(TagTransformModel.java:431)
> at
> freemarker.ext.jsp.TagTransformModel$TagWriter.afterBody(TagTransformModel.java:407)
> at freemarker.core.Environment.visit(Environment.java:299)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.visit(Environment.java:297)
> at freemarker.core.UnifiedCall.accept(UnifiedCall.java:130)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.MixedContent.accept(MixedContent.java:92)
> at freemarker.core.Environment.visit(Environment.java:208)
> at freemarker.core.Environment.process(Environment.java:188)
> at freemarker.template.Template.process(Template.java:237)
> at
> org.apache.struts2.views.freemarker.FreemarkerResult.doExecute(FreemarkerResult.java:168)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
