Yup. It only takes one AuthenticationHandler hung thread to block
further requests from even trying to authenticate.
Opened a JIRA and attached a simple Maven Overlay demonstrating the issue.
https://issues.jasig.org/browse/CAS-993
I'll move discussion of what to do about this over to cas-dev.
This tends to support a hypothesis that when this conversation deadlock
happened last week, it was an AuthenticationHandler that had started
responding really slowly, thereby hanging on to this lock for a long time.
Andrew
Ok. I think I can make this exception happen by introducing a custom
AuthenticationHandler that simply hangs forever.
At least, I started getting this in a terribly messy log at the TRACE
level. ootb cas-server 3.4.8 with only the custom loop-forever
authentication handler introduced.
[
2011-06-20 16:13:02,558 TRACE
[org.jasig.cas.web.NoSuchFlowExecutionExceptionResolver] - Entering
method [resolveException with arguments
[[org.apache.catalina.connector.RequestFacade@5889949a,
org.apache.catalina.connector.ResponseFacade@307b37df,
[FlowHandlerMapping.DefaultFlowHandler@da3a52c],
org.springframework.webflow.conversation.impl.LockTimeoutException:
Unable to acquire conversation lock after 30 seconds]]
2011-06-20 16:13:02,558 TRACE
[org.jasig.cas.web.NoSuchFlowExecutionExceptionResolver] - Leaving
method [resolveException] with return value [null].
]
Looks to me like it only hangs one thread and no other threads manage
to even get to the authentication handler, though I submitted a
handful of login form posts from a couple of browsers.
I'll gin up a quick HangsJustOneThreadForeverAuthenticationHandler to
prove that it only takes one.
Andrew
Math, hi.
Our stuff is running on OSX server. I've seen several mysql databases
(we put some log entries into mysql) stop accepting new entries when
the db gets to the 10Gb mark. No limit imposed by us, just
empirically observed.
My wild guess is that maybe the inability to insert new records at
that point caused a deadlock.
We're using queries for the audit data from Inspektr. Saved queries
are handy. There is no built in managment gui for the data, but I
imagine Crystal Reports would be handy.
Best,
KC
On Feb 14, 2011, at 11:38 AM, Matthew Selwood wrote:
Hi Kim,
I’m in the process of implemented Inspektr on our CAS build here.
What 10Gb threshold are you referred to? Is this a DB limit you’ve
put in place?
On a similar note, is there a ‘user friendly’ way of viewing the
database audit and stats, or do you just write queries?
Thanks,
Matt
-------------------------------
Matthew Selwood
Programmer Analyst – Web Services
University of Victoria
250.472.5565 | [email protected] | CLE C006
On 2/13/11 9:36 PM, "Cary, Kim"<[email protected]> wrote:
Our CAS service came up unresponsive (connection timed out) after
throwing about 6 of these errors tonite.
Google doesn't have much on this, but one case shows some
involvement with c3po failing to talk to the database. We use
inspektr and our audit database hit a 10Gb threshold tonite.
Restarting tomcat without making any other changes put the service
back into operation.
Can anyone share any clues on pursuing the root cause of the outage?
Thanks,
KC
Feb 13, 2011 7:12:03 PM
org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet cas threw exception
org.springframework.webflow.conversation.impl.LockTimeoutException:
Unable to acquire conversation lock after 30 seconds
at
org.springframework.webflow.conversation.impl.JdkConcurrentConversationLock.lock(JdkConcurrentConversationLock.java:44)
at
org.springframework.webflow.conversation.impl.ContainedConversation.lock(ContainedConversation.java:69)
at
org.springframework.webflow.execution.repository.support.ConversationBackedFlowExecutionLock.lock(ConversationBackedFlowExecutionLock.java:51)
at
org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:160)
at
org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:563)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody2(SafeDispatcherServlet.java:115)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody3$advice(SafeDispatcherServlet.java:44)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:1)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:46)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:613)
Feb 13, 2011 7:12:14 PM
org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet cas threw exception
org.springframework.webflow.conversation.impl.LockTimeoutException:
Unable to acquire conversation lock after 30 seconds
at
org.springframework.webflow.conversation.impl.JdkConcurrentConversationLock.lock(JdkConcurrentConversationLock.java:44)
at
org.springframework.webflow.conversation.impl.ContainedConversation.lock(ContainedConversation.java:69)
at
org.springframework.webflow.execution.repository.support.ConversationBackedFlowExecutionLock.lock(ConversationBackedFlowExecutionLock.java:51)
at
org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:160)
at
org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:563)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody2(SafeDispatcherServlet.java:115)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody3$advice(SafeDispatcherServlet.java:44)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:1)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:46)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:613)
--
You are currently subscribed to [email protected] as:
[email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/cas-user
--
You are currently subscribed to [email protected] as:
[email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/cas-user
