Re: [cas-user] CAS ---> Memcached Intermittent errors

[venu.alla]

Scott,

We ran into a boad loat of "Connection timed out" issues starting last 
night and we had to restart cas server to return to normal state. CAS 
was essentially unfunctional during this state.

We are hosting cas and memcache on its own server, OS is RH.

Our deployment is as follows.

cas ---> LB ----> memcache[1,2] with repcache.

What is your env and deployment  like?
Venu

Here is the stack trace we saw upon login attempt.....


*message*

*description* The server encountered an internal error () that prevented 
it from fulfilling this request.

*exception*

org.springframework.web.util.NestedServletException: Request processing 
failed; nested exception is 
org.springframework.webflow.engine.ActionExecutionException: Exception 
thrown executing [annotatedact...@67e90fee targetAction = 
org.jasig.cas.web.flow.generateserviceticketact...@7615078d, attributes 
= map[[empty]]] in state 'generateServiceTicket' of flow 'login-webflow' 
-- action execution attributes were 'map[[empty]]'; nested exception is 
net.spy.memcached.OperationTimeoutException: Timeout waiting for value

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

*root cause*

org.springframework.webflow.engine.ActionExecutionException: Exception 
thrown executing [annotatedact...@67e90fee targetAction = 
org.jasig.cas.web.flow.generateserviceticketact...@7615078d, attributes 
= map[[empty]]] in state 'generateServiceTicket' of flow 'login-webflow' 
-- action execution attributes were 'map[[empty]]'; nested exception is 
net.spy.memcached.OperationTimeoutException: Timeout waiting for value

org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:64)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

*root cause*

net.spy.memcached.OperationTimeoutException: Timeout waiting for value
    net.spy.memcached.MemcachedClient.get(MemcachedClient.java:771)
    net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)

org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119)

org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35)

org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173)

org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245)

org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)

org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)

org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)

org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

*root cause*

java.util.concurrent.TimeoutException: Timed out waiting for operation

net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1437)
    net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
    net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)

org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119)

org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35)

org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173)

org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245)

org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)

org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)

org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)

org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
    org.springframework.webflow.engine.State.enter(State.java:191)

org.springframework.webflow.engine.Transition.execute(Transition.java:212)

org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
    org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

*note* The full stack trace of the root cause is available in the Apache 
Tomcat/6.0.18 logs.




Scott Battaglia wrote:
I just checked our logs and we've only encountered connection reset by 
peer 10 times since 11/14/2008.  Is there any specific setting you want 
me to check for you?

Cheers,
Scott


On Mon, Jul 20, 2009 at 1:36 PM, venu.alla <venu.a...@berkeley.edu 
<mailto:venu.a...@berkeley.edu>> wrote:

    hi,
    We are currently testing Memcached(with repcached) as CAS Service
    Ticket Registry. We see the following errors intermittently. Any one
    else seen these in their own env and if so, any feed back as to why
    and how to reduce them. We also notice that, when timedout error
    occurs, the exception is thrown all the way up into tomcat which in
    turn shows up on end user page.

    1) Connection Timed out (rare but impacts service availability)
    2) Connection Reset by Peer ( more frequently than type-1)
     >>>>>>>TIMED OUT>>>>>>>>>>>>>>>
    2009-07-20 08:26:50.385 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting due to exception on {QA
    sa=xxxx.berkeley.edu/x.xx.xx.xx:11211
    <http://xxxx.berkeley.edu/x.xx.xx.xx:11211>, #Rops=28, #Wops=0,
    #iq=0,
    toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77,
    topWop=null, toWrite=0, interested=1}
    java.io.IOException: Connection timed out
           at sun.nio.ch.FileDispatcher.read0(Native Method)
           at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
           at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
           at sun.nio.ch.IOUtil.read(IOUtil.java:206)
           at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
           at
    
net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294)
           at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262)
           at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
           at
    net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
    2009-07-20 08:26:50.386 WARN net.spy.memcached.MemcachedConnection:
    Closing, and reopening {QA sa=xxxx.berkeley.edu/x.x.x.x:11211
    <http://xxxx.berkeley.edu/x.x.x.x:11211>, #Rops=28, #Wops=0, #iq=0,
    toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77,
    topWop=null, toWrite=0, interested=1}, attempt 0.
    2009-07-20 08:26:50.387 WARN
    net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl:  Discarding
    partially completed op:
    Note: Many discards after this before it returned to normal state.

     >>>>>>>>>>>> Connection RESET >>>>>>>>>>>>>>>>>>>>>>>>
    2009-07-20 00:38:01.390 INFO net.spy.memcached.MemcachedConnection:
    Reconnecting due to exception on {QA sa=xx.x.edu/x.x.x.x:11211
    <http://xx.x.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0,
    toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca,
    topWop=null, toWrite=0, interested=1}
    java.io.IOException: Connection reset by peer
           at sun.nio.ch.FileDispatcher.read0(Native Method)
           at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
           at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
           at sun.nio.ch.IOUtil.read(IOUtil.java:206)
           at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
           at
    
net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294)
           at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262)
           at
    net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
           at
    net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
    2009-07-20 00:38:01.390 WARN net.spy.memcached.MemcachedConnection:
    Closing, and reopening {QA sa=xxx.berkeley.edu/x.x.x.x:11211
    <http://xxx.berkeley.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0,
    toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca,
    topWop=null, toWrite=0, interested=1}, attempt 0.


    -- 
    You are currently subscribed to cas-user@lists.jasig.org
    <mailto:cas-user@lists.jasig.org> as: scott.battag...@gmail.com
    <mailto:scott.battag...@gmail.com>
    To unsubscribe, change settings or access archives, see
    http://www.ja-sig.org/wiki/display/JSG/cas-user


--
You are currently subscribed to cas-user@lists.jasig.org as: 
venu.a...@berkeley.edu
To unsubscribe, change settings or access archives, see 
http://www.ja-sig.org/wiki/display/JSG/cas-user


--
You are currently subscribed to cas-user@lists.jasig.org as: 
arch...@mail-archive.com
To unsubscribe, change settings or access archives, see 
http://www.ja-sig.org/wiki/display/JSG/cas-user