We're essentially running Load Balancer -> CAS (x2) Each of the CAS machines has a memcache instance on it and CAS refers to the internal name of the memache server (it does not go through the load balancer).
On Thu, Jul 23, 2009 at 2:24 PM, venu.alla <[email protected]> wrote: > Scott, > > We ran into a boad loat of "Connection timed out" issues starting last > night and we had to restart cas server to return to normal state. CAS was > essentially unfunctional during this state. > > We are hosting cas and memcache on its own server, OS is RH. > > Our deployment is as follows. > > cas ---> LB ----> memcache[1,2] with repcache. > > What is your env and deployment like? > Venu > > Here is the stack trace we saw upon login attempt..... > > > *message* > > *description* The server encountered an internal error () that prevented it > from fulfilling this request. > > *exception* > > org.springframework.web.util.NestedServletException: Request processing > failed; nested exception is > org.springframework.webflow.engine.ActionExecutionException: Exception > thrown executing [annotatedact...@67e90fee targetAction = > org.jasig.cas.web.flow.generateserviceticketact...@7615078d, attributes = > map[[empty]]] in state 'generateServiceTicket' of flow 'login-webflow' -- > action execution attributes were 'map[[empty]]'; nested exception is > net.spy.memcached.OperationTimeoutException: Timeout waiting for value > > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583) > > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) > javax.servlet.http.HttpServlet.service(HttpServlet.java:637) > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) > > > org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > > *root cause* > > org.springframework.webflow.engine.ActionExecutionException: Exception > thrown executing [annotatedact...@67e90fee targetAction = > org.jasig.cas.web.flow.generateserviceticketact...@7615078d, attributes = > map[[empty]]] in state 'generateServiceTicket' of flow 'login-webflow' -- > action execution attributes were 'map[[empty]]'; nested exception is > net.spy.memcached.OperationTimeoutException: Timeout waiting for value > > > org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:64) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) > > > org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) > > > org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) > > > org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) > > > org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) > > > org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) > > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) > > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) > > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) > > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) > javax.servlet.http.HttpServlet.service(HttpServlet.java:637) > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) > > > org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > > *root cause* > > net.spy.memcached.OperationTimeoutException: Timeout waiting for value > net.spy.memcached.MemcachedClient.get(MemcachedClient.java:771) > net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784) > > > org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119) > > > org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35) > > > org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173) > > > org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245) > > > org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37) > > > org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192) > > > org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146) > > > org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) > > > org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) > > > org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) > > > org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) > > > org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) > > > org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) > > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) > > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) > > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) > > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) > javax.servlet.http.HttpServlet.service(HttpServlet.java:637) > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) > > > org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > > *root cause* > > java.util.concurrent.TimeoutException: Timed out waiting for operation > > > net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1437) > net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764) > net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784) > > > org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119) > > > org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35) > > > org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173) > > > org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245) > > > org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37) > > > org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192) > > > org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146) > > > org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > org.springframework.webflow.engine.State.enter(State.java:191) > > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > > > org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) > > > org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) > > > org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) > > > org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) > > > org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) > > > org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) > > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) > > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) > > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) > > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) > javax.servlet.http.HttpServlet.service(HttpServlet.java:637) > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) > > > org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) > > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > > *note* The full stack trace of the root cause is available in the Apache > Tomcat/6.0.18 logs. > > > > > Scott Battaglia wrote: > >> I just checked our logs and we've only encountered connection reset by >> peer 10 times since 11/14/2008. Is there any specific setting you want me >> to check for you? >> >> Cheers, >> Scott >> >> >> On Mon, Jul 20, 2009 at 1:36 PM, venu.alla <[email protected]<mailto: >> [email protected]>> wrote: >> >> hi, >> We are currently testing Memcached(with repcached) as CAS Service >> Ticket Registry. We see the following errors intermittently. Any one >> else seen these in their own env and if so, any feed back as to why >> and how to reduce them. We also notice that, when timedout error >> occurs, the exception is thrown all the way up into tomcat which in >> turn shows up on end user page. >> >> 1) Connection Timed out (rare but impacts service availability) >> 2) Connection Reset by Peer ( more frequently than type-1) >> >>>>>>>TIMED OUT>>>>>>>>>>>>>>> >> 2009-07-20 08:26:50.385 INFO net.spy.memcached.MemcachedConnection: >> Reconnecting due to exception on {QA >> sa=xxxx.berkeley.edu/x.xx.xx.xx:11211 >> <http://xxxx.berkeley.edu/x.xx.xx.xx:11211>, #Rops=28, #Wops=0, >> #iq=0, >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77, >> topWop=null, toWrite=0, interested=1} >> java.io.IOException: Connection timed out >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:206) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> at >> >> >> net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) >> at >> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) >> 2009-07-20 08:26:50.386 WARN net.spy.memcached.MemcachedConnection: >> Closing, and reopening {QA sa=xxxx.berkeley.edu/x.x.x.x:11211 >> <http://xxxx.berkeley.edu/x.x.x.x:11211>, #Rops=28, #Wops=0, #iq=0, >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77, >> topWop=null, toWrite=0, interested=1}, attempt 0. >> 2009-07-20 08:26:50.387 WARN >> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding >> partially completed op: >> Note: Many discards after this before it returned to normal state. >> >> >>>>>>>>>>>> Connection RESET >>>>>>>>>>>>>>>>>>>>>>>> >> 2009-07-20 00:38:01.390 INFO net.spy.memcached.MemcachedConnection: >> Reconnecting due to exception on {QA sa=xx.x.edu/x.x.x.x:11211 >> <http://xx.x.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0, >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca, >> topWop=null, toWrite=0, interested=1} >> java.io.IOException: Connection reset by peer >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:206) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> at >> >> >> net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) >> at >> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) >> 2009-07-20 00:38:01.390 WARN net.spy.memcached.MemcachedConnection: >> Closing, and reopening {QA sa=xxx.berkeley.edu/x.x.x.x:11211 >> <http://xxx.berkeley.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0, >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca, >> topWop=null, toWrite=0, interested=1}, attempt 0. >> >> >> -- You are currently subscribed to [email protected] >> <mailto:[email protected]> as: [email protected] >> <mailto:[email protected]> >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> >> -- >> You are currently subscribed to [email protected] as: >> [email protected] >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> > -- > You are currently subscribed to [email protected] as: > [email protected] > To unsubscribe, change settings or access archives, see > http://www.ja-sig.org/wiki/display/JSG/cas-user > -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
