Hi, I have CAS 3.4.11 running on a single server Tomcat environment and have it configured to store tickets in a Microsoft SQL Server database using JPATicketRegistry. I've been running load tests against the login and logout calls of the CAS server, and I kept seeing a lot of log entries for deadlocking, so I made the suggested changes:
1. Implemented the Indexes on the two tables as suggested on this page: https://wiki.jasig.org/display/CASUM/JpaTicketRegistry#JpaTicketRegistry-AvoidingDeadlocks - Deadlocking still occurred 2. Disabled the ticketRegistryCleaner - Deadlocking still occurred 3. Updated the JpaTicketRegistry.java file and commented out the Pessimistic Lock as seen on this page: https://issues.jasig.org/browse/CAS-1051- Deadlocking still occurred The database server isn't struggling with performance, as it is a high spec. I have also performed Load testing whilst running the Performance Monitor on the SQL Server, which was configured to log Locks and transactions on the database, which showed the deadlocking reports. The load test is only 500 users over 20 minutes, so it's certainly not a huge rush of users! Here are the deadlocking log entries: Both login and logout errors occur. 2012-09-20 15:02:16,281 ERROR [org.hibernate.util.JDBCExceptionReporter] - <Transaction (Process ID 61) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.> 2012-09-20 15:02:16,281 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] - <Could not synchronize database state with session> org.hibernate.exception.LockAcquisitionException: could not update: [org.jasig.cas.ticket.TicketGrantingTicketImpl#TGT-68-XXXXXX-cas] at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:107) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2594) at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2476) at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2803) at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216) at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:383) at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:133) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:76) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:467) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy34.grantServiceTicket(Unknown Source) at org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:39) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) at org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:77) at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) at org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145) at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:51) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:391) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:386) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:391) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:386) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105) at org.springframework.webflow.engine.State.enter(State.java:194) at org.springframework.webflow.engine.Transition.execute(Transition.java:227) at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:391) at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:386) at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) at org.springframework.webflow.engine.ViewState.handleEvent(ViewState.java:254) at org.springframework.webflow.engine.ViewState.resume(ViewState.java:218) at org.springframework.webflow.engine.Flow.resume(Flow.java:545) at org.springframework.webflow.engine.impl.FlowExecutionImpl.resume(FlowExecutionImpl.java:259) at org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:169) at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560) at javax.servlet.http.HttpServlet.service(HttpServlet.java:641) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody2(SafeDispatcherServlet.java:115) at org.jasig.cas.web.init.SafeDispatcherServlet.service_aroundBody3$advice(SafeDispatcherServlet.java:44) at org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:1) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1805) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.sql.SQLException: Transaction (Process ID 61) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368) at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2820) at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2258) at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:632) at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584) at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546) at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:504) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:46) at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2572) ... 99 more My deployerConfigContext.xml uses the SearchModeSearchDatabaseAuthenticationHandler: <bean class="org.jasig.cas.adaptors.jdbc.SearchModeSearchDatabaseAuthenticationHandler"> <property name="tableUsers"><value>viewToSearch</value></property> <property name="fieldUser"><value>EmailAddress</value></property> <property name="fieldPassword"><value>Password</value></property> <property name="dataSource" ref="sqlIntegraServerDataSource"/> </bean> And the dataSource is: <bean id="sqlIntegraServerDataSource" class="org.apache.commons.dbcp.BasicDataSource" p:driverClassName="net.sourceforge.jtds.jdbc.Driver" p:url="jdbc:jtds:sqlserver://IP_ADDRESS:PORT;databaseName=DB_NAME" p:password="SOME_PASSWORD" p:username="SOME_USER" /> Are there any other configuration changes I could make which will help reduce or eliminate these deadlocks please? Thanks, Nathan -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
