Hi Ktc, Thanks for this detailed email with lots of details and hard questions!
This is not an easy mail to answer and I certainly don’t have the answers. You’re the first one to report such issue AFAIK and that’s most certainly due to the scale at which you use XWiki :) Let’s see if someone has the answer here. If you don’t get any answer, know that there’s an option for you which is to contract a company offering dedicated professional support for XWiki and have them spend quality time debugging this with you (see http://www.xwiki.org/xwiki/bin/view/Main/Support#HProfessionalSupport). If you find the solution, some Pull Requests would be much appreciated to fix the problems. Thanks -Vincent > On 25 Oct 2017, at 18:31, ktc <[email protected]> wrote: > > We ran into an issue recently where users were not able to access the Wiki > due to the XWiki security cache holding up access to read requests as we > were experiencing heap exhaustion. A thread dump of a host showed that there > were 211 security cache read locks and 2 security cache write locks during > the event. Read locks are supposedly parallelizable, so we suspect that it > was the write locks that blocked the read requests. > > Looking into the write locks, we found > *DefaultSecurityCacheRulesInvalidatorListener* shares a read-write lock > (*SecurityCacheRulesInvalidator*) with the security cache loader > (*DefaultSecurityCacheLoader*). It appears that these write requests to the > security cache were potentially creating a deadlock, blocking any further > read requests. > > Our investigation of this event resulted in the following questions that we > are hoping to get answered by reaching out here: > > 1. What exactly happens in the event of a write failure to the Security > Cache? Are we correct in understanding that the write request will continue > to hold a lock on the cache, preventing all read requests from proceeding? > 2. If so, are there any mitigations around the global read-write lock on the > security cache? Or, perhaps, is the best action is to simply move toward a > more scalable caching mechanism (i.e. a centralized cache such as Elastic > Cache)? > 3. It looks as if all threads doing a read/write request to the > SecurityCache were all parked waiting for the same thread (3081f946); > however, the thread is not showing up in the thread dump so we are not sure > what it is stuck on. Does the XWiki team have any suspicions as to what > could be hold up the requests to the SecurityCache? > > Thank you! > > --- > > For more information on the event: > > From our investigation, we concluded that the outage happened because: > > 1. Remote events triggered a security cache update and requested a couple of > write locks > 2. The writes could not execute (perhaps due to the heap exhaustion, > although root cause of write failure is unknown) > 3. ReentrantReadWriteLock lock followed a fair policy, which releases locks > in time order, so read requests were forced to wait for any previous write > requests to finish. Because the write requests couldn't finish, the > remaining 211 threads were blocked. > > * Possible remote events that trigger security cache update: changes to an > XWiki group such as leaving the admin group, user first time visiting XWiki, > etc.. > > ---- > > *LDAP Connection Lock example:* > > Thread t@225108 > java.lang.Thread.State: TIMED_WAITING > at java.lang.Object.wait(Native Method) > - waiting on <7249fc39> (a com.novell.ldap.Connection) > at com.novell.ldap.Connection.waitForReader(Unknown Source) > at com.novell.ldap.Connection.startReader(Unknown Source) > at com.novell.ldap.Connection.connect(Unknown Source) > at com.novell.ldap.Connection.connect(Unknown Source) > at com.novell.ldap.LDAPConnection.connect(Unknown Source) > at > com.ownership.groups.ldap.DefaultLdapConnection.connect(DefaultLdapConnection.java:25) > at > com.ownership.groups.ldap.LdapClient.queryGroups(LdapClient.java:99) > at > com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:64) > at > com.ownership.groups.ldap.LdapClient.getGroupsForUser(LdapClient.java:59) > at > com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:113) > - locked <7cc1cfca> (a java.lang.String) > at > com.ownership.groups.GroupService.getAllGroupsReferencesForMember(GroupService.java:75) > at > org.xwiki.security.internal.DefaultUserBridge.getGroupsReferencesFor(DefaultUserBridge.java:122) > at > org.xwiki.security.internal.DefaultUserBridge.getAllGroupsFor(DefaultUserBridge.java:78) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:350) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserGroups(DefaultSecurityCacheLoader.java:342) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadUserEntry(DefaultSecurityCacheLoader.java:318) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadAccessEntries(DefaultSecurityCacheLoader.java:221) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.loadRequiredEntries(DefaultSecurityCacheLoader.java:193) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:148) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109) > at > org.xwiki.display.internal.AbstractDocumentTitleDisplayer.displayTitle(AbstractDocumentTitleDisplayer.java:163) > at > org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:149) > at > org.xwiki.display.internal.AbstractDocumentTitleDisplayer.display(AbstractDocumentTitleDisplayer.java:58) > at > org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:96) > at > org.xwiki.display.internal.DefaultDocumentDisplayer.display(DefaultDocumentDisplayer.java:39) > at > org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:123) > at > org.xwiki.sheet.internal.SheetDocumentDisplayer.display(SheetDocumentDisplayer.java:52) > at > org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:68) > at > org.xwiki.display.internal.ConfiguredDocumentDisplayer.display(ConfiguredDocumentDisplayer.java:42) > at > com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1538) > at > com.xpn.xwiki.doc.XWikiDocument.getRenderedTitle(XWikiDocument.java:1558) > at com.xpn.xwiki.api.Document.getDisplayTitle(Document.java:345) > at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.runtime.parser.node.PropertyExecutor.execute(PropertyExecutor.java:142) > at > org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:539) > at > org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:198) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.apache.velocity.runtime.parser.node.ASTStringLiteral.value(ASTStringLiteral.java:330) > at > org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71) > at > org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) > at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) > at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) > at > org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) > at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) > at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) > at > org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180) > at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463) > at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210) > at > org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425) > at > org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228) > at > org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) > at > org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215) > at > com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176) > at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) > at > org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) > - locked <1cc68e56> (a org.apache.tomcat.util.net.SocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - locked <79120011> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > ---- > > *Security Cache Read Lock example:* > > Thread t@225106 > java.lang.Thread.State: WAITING > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <3081f946> (a > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) > at > org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidator.suspend(DefaultSecurityCacheRulesInvalidator.java:56) > at > org.xwiki.security.authorization.cache.internal.DefaultSecurityCacheLoader.load(DefaultSecurityCacheLoader.java:144) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.getAccess(DefaultAuthorizationManager.java:218) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.evaluateSecurityAccess(DefaultAuthorizationManager.java:159) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.hasSecurityAccess(DefaultAuthorizationManager.java:152) > at > org.xwiki.security.authorization.DefaultAuthorizationManager.hasAccess(DefaultAuthorizationManager.java:109) > at > org.xwiki.security.authorization.internal.XWikiCachingRightService.hasAccessLevel(XWikiCachingRightService.java:309) > at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:270) > at com.xpn.xwiki.api.XWiki.getDocument(XWiki.java:254) > at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567) > at > org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71) > at > org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) > at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) > at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) > at > org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) > at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) > at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) > at > org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.XWiki.parseTemplate(XWiki.java:2080) > at com.xpn.xwiki.api.XWiki.parseTemplate(XWiki.java:886) > at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395) > at > org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384) > at > org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173) > at > org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) > at > org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:216) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:311) > at > org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:230) > at > org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207) > at > org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106) > at > org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluateInternal(DefaultVelocityEngine.java:259) > at > org.xwiki.velocity.internal.DefaultVelocityEngine.evaluate(DefaultVelocityEngine.java:222) > at > com.xpn.xwiki.render.DefaultVelocityManager.evaluate(DefaultVelocityManager.java:361) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.evaluateContent(InternalTemplateManager.java:790) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:667) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:645) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.renderFromSkin(InternalTemplateManager.java:616) > at > com.xpn.xwiki.internal.template.InternalTemplateManager.render(InternalTemplateManager.java:602) > at > com.xpn.xwiki.internal.template.DefaultTemplateManager.render(DefaultTemplateManager.java:77) > at com.xpn.xwiki.XWiki.evaluateTemplate(XWiki.java:2102) > at com.xpn.xwiki.web.Utils.parseTemplate(Utils.java:180) > at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:463) > at com.xpn.xwiki.web.XWikiAction.execute(XWikiAction.java:210) > at > org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425) > at > org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228) > at > org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913) > at > org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.xpn.xwiki.web.ActionFilter.doFilter(ActionFilter.java:112) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.wysiwyg.server.filter.ConversionFilter.doFilter(ConversionFilter.java:127) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > actiontrace.generation.servlet.StorageBindingFilter.doFilter(StorageBindingFilter.java:40) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.spnego.http.InitiateAction.run(HttpAuthFilter.java:215) > at > com.spnego.http.HttpAuthFilter.executeFilterWithCreds(HttpAuthFilter.java:176) > at com.spnego.http.HttpAuthFilter.doFilter(HttpAuthFilter.java:74) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at com.build.brazil.filters.FQDNFilter.doFilter(FQDNFilter.java:170) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SetHTTPHeaderFilter.doFilter(SetHTTPHeaderFilter.java:63) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SavedRequestRestorerFilter.doFilter(SavedRequestRestorerFilter.java:208) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.container.servlet.filters.internal.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:111) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.xwiki.resource.servlet.RoutingFilter.doFilter(RoutingFilter.java:136) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > at com.tomcat.valves.QueryLogValve.invoke(QueryLogValve.java:157) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) > at > org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1126) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) > - locked <4da86584> (a org.apache.tomcat.util.net.SocketWrapper) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - locked <1479d987> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > ---- > > *Security Cache Write Lock example:* > > Thread t@178 > java.lang.Thread.State: WAITING > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <3081f946> (a > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) > at > org.xwiki.security.authorization.internal.DefaultSecurityCacheRulesInvalidatorListener.onEvent(DefaultSecurityCacheRulesInvalidatorListener.java:186) > at > org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:304) > at > org.xwiki.observation.internal.DefaultObservationManager.notify(DefaultObservationManager.java:269) > at > org.xwiki.observation.remote.internal.DefaultRemoteObservationManager.notify(DefaultRemoteObservationManager.java:168) > at > com.wiki.observation.remote.jgroups.JGroupsReceiver.receive(JGroupsReceiver.java:87) > at org.jgroups.JChannel.up(JChannel.java:768) > at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1034) > at org.jgroups.protocols.FRAG2.up(FRAG2.java:182) > at org.jgroups.protocols.FlowControl.up(FlowControl.java:438) > at org.jgroups.stack.Protocol.up(Protocol.java:426) > at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294) > at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:487) > at > org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:989) > at > org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:919) > at > org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:851) > at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611) > at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155) > at org.jgroups.protocols.FD.up(FD.java:260) > at org.jgroups.protocols.MERGE3.up(MERGE3.java:292) > at org.jgroups.protocols.Discovery.up(Discovery.java:296) > at org.jgroups.protocols.TP.passMessageUp(TP.java:1657) > at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - locked <2ea19041> (a > java.util.concurrent.ThreadPoolExecutor$Worker) > > > > -- > Sent from: http://xwiki.475771.n2.nabble.com/XWiki-Dev-f475773.html > >
