Re: [cas-user] CAS ---> Memcached Intermittent errors

[venu.alla]

Scott,

What versions of CAS and Memcache+Repcached are your running and what is 
the OS?

Here is what we are running
CAS: 3.3.1
memcached-1.2.8-repcached-2.2.tar.gz

Venu

Scott Battaglia wrote:
We're essentially running

Load Balancer -> CAS (x2)  Each of the CAS machines has a memcache 
instance on it and CAS refers to the internal name of the memache server 
(it does not go through the load balancer).



On Thu, Jul 23, 2009 at 2:24 PM, venu.alla <venu.a...@berkeley.edu 
<mailto:venu.a...@berkeley.edu>> wrote:

    Scott,

    We ran into a boad loat of "Connection timed out" issues starting
    last night and we had to restart cas server to return to normal
    state. CAS was essentially unfunctional during this state.

    We are hosting cas and memcache on its own server, OS is RH.

    Our deployment is as follows.

    cas ---> LB ----> memcache[1,2] with repcache.

    What is your env and deployment  like?
    Venu

    Here is the stack trace we saw upon login attempt.....


    *message*

    *description* The server encountered an internal error () that
    prevented it from fulfilling this request.

    *exception*

    org.springframework.web.util.NestedServletException: Request
    processing failed; nested exception is
    org.springframework.webflow.engine.ActionExecutionException:
    Exception thrown executing [annotatedact...@67e90fee targetAction =
    org.jasig.cas.web.flow.generateserviceticketact...@7615078d,
    attributes = map[[empty]]] in state 'generateServiceTicket' of flow
    'login-webflow' -- action execution attributes were 'map[[empty]]';
    nested exception is net.spy.memcached.OperationTimeoutException:
    Timeout waiting for value

    
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583)

    
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

    
org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

    
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

    *root cause*

    org.springframework.webflow.engine.ActionExecutionException:
    Exception thrown executing [annotatedact...@67e90fee targetAction =
    org.jasig.cas.web.flow.generateserviceticketact...@7615078d,
    attributes = map[[empty]]] in state 'generateServiceTicket' of flow
    'login-webflow' -- action execution attributes were 'map[[empty]]';
    nested exception is net.spy.memcached.OperationTimeoutException:
    Timeout waiting for value

    
org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:64)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    
org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

    
org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

    
org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

    
org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

    
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

    
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

    
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

    
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

    
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

    
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

    
org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

    
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

    *root cause*

    net.spy.memcached.OperationTimeoutException: Timeout waiting for value
       net.spy.memcached.MemcachedClient.get(MemcachedClient.java:771)
       net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)

    
org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119)

    
org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35)

    
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173)

    
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245)

    
org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)

    
org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)

    
org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)

    
org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    
org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

    
org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

    
org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

    
org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

    
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

    
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

    
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

    
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

    
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

    
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

    
org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

    
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

    *root cause*

    java.util.concurrent.TimeoutException: Timed out waiting for operation

    
net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1437)
       net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
       net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)

    
org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119)

    
org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35)

    
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173)

    
org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245)

    
org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)

    
org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)

    
org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)

    
org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
       org.springframework.webflow.engine.State.enter(State.java:191)

    org.springframework.webflow.engine.Transition.execute(Transition.java:212)

    
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
       org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)

    
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)

    
org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)

    
org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)

    
org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)

    
org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)

    
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)

    
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

    
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)

    
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

    
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

    
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
       javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

    
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)

    
org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)

    
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

    *note* The full stack trace of the root cause is available in the
    Apache Tomcat/6.0.18 logs.




    Scott Battaglia wrote:

        I just checked our logs and we've only encountered connection
        reset by peer 10 times since 11/14/2008.  Is there any specific
        setting you want me to check for you?

        Cheers,
        Scott


        On Mon, Jul 20, 2009 at 1:36 PM, venu.alla
        <venu.a...@berkeley.edu <mailto:venu.a...@berkeley.edu>
        <mailto:venu.a...@berkeley.edu <mailto:venu.a...@berkeley.edu>>>
        wrote:

           hi,
           We are currently testing Memcached(with repcached) as CAS Service
           Ticket Registry. We see the following errors intermittently.
        Any one
           else seen these in their own env and if so, any feed back as
        to why
           and how to reduce them. We also notice that, when timedout error
           occurs, the exception is thrown all the way up into tomcat
        which in
           turn shows up on end user page.

           1) Connection Timed out (rare but impacts service availability)
           2) Connection Reset by Peer ( more frequently than type-1)
            >>>>>>>TIMED OUT>>>>>>>>>>>>>>>
           2009-07-20 08:26:50.385 INFO
        net.spy.memcached.MemcachedConnection:
           Reconnecting due to exception on {QA
           sa=xxxx.berkeley.edu/x.xx.xx.xx:11211
        <http://xxxx.berkeley.edu/x.xx.xx.xx:11211>
           <http://xxxx.berkeley.edu/x.xx.xx.xx:11211>, #Rops=28, #Wops=0,

           #iq=0,
         
         toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77,
           topWop=null, toWrite=0, interested=1}
           java.io.IOException: Connection timed out
                  at sun.nio.ch.FileDispatcher.read0(Native Method)
                  at
        sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
                  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
                  at sun.nio.ch.IOUtil.read(IOUtil.java:206)
                  at
        sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
                  at
         
         net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294)
                  at
         
         net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262)
                  at
         
         net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
                  at
           net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
           2009-07-20 08:26:50.386 WARN
        net.spy.memcached.MemcachedConnection:
           Closing, and reopening {QA sa=xxxx.berkeley.edu/x.x.x.x:11211
        <http://xxxx.berkeley.edu/x.x.x.x:11211>
           <http://xxxx.berkeley.edu/x.x.x.x:11211>, #Rops=28, #Wops=0,
        #iq=0,

         
         toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77,
           topWop=null, toWrite=0, interested=1}, attempt 0.
           2009-07-20 08:26:50.387 WARN
           net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl:
         Discarding
           partially completed op:
           Note: Many discards after this before it returned to normal
        state.

            >>>>>>>>>>>> Connection RESET >>>>>>>>>>>>>>>>>>>>>>>>
           2009-07-20 00:38:01.390 INFO
        net.spy.memcached.MemcachedConnection:
           Reconnecting due to exception on {QA
        sa=xx.x.edu/x.x.x.x:11211 <http://xx.x.edu/x.x.x.x:11211>
           <http://xx.x.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0,

         
         toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca,
           topWop=null, toWrite=0, interested=1}
           java.io.IOException: Connection reset by peer
                  at sun.nio.ch.FileDispatcher.read0(Native Method)
                  at
        sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
                  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
                  at sun.nio.ch.IOUtil.read(IOUtil.java:206)
                  at
        sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
                  at
         
         net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294)
                  at
         
         net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262)
                  at
         
         net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
                  at
           net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
           2009-07-20 00:38:01.390 WARN
        net.spy.memcached.MemcachedConnection:
           Closing, and reopening {QA sa=xxx.berkeley.edu/x.x.x.x:11211
        <http://xxx.berkeley.edu/x.x.x.x:11211>
           <http://xxx.berkeley.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0,

         
         toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca,
           topWop=null, toWrite=0, interested=1}, attempt 0.


           --    You are currently subscribed to
        cas-user@lists.jasig.org <mailto:cas-user@lists.jasig.org>
           <mailto:cas-user@lists.jasig.org
        <mailto:cas-user@lists.jasig.org>> as: scott.battag...@gmail.com
        <mailto:scott.battag...@gmail.com>
           <mailto:scott.battag...@gmail.com
        <mailto:scott.battag...@gmail.com>>

           To unsubscribe, change settings or access archives, see
           http://www.ja-sig.org/wiki/display/JSG/cas-user


        -- 
        You are currently subscribed to cas-user@lists.jasig.org
        <mailto:cas-user@lists.jasig.org> as: venu.a...@berkeley.edu
        <mailto:venu.a...@berkeley.edu>

        To unsubscribe, change settings or access archives, see
        http://www.ja-sig.org/wiki/display/JSG/cas-user


    -- 
    You are currently subscribed to cas-user@lists.jasig.org
    <mailto:cas-user@lists.jasig.org> as: scott.battag...@gmail.com
    <mailto:scott.battag...@gmail.com>
    To unsubscribe, change settings or access archives, see
    http://www.ja-sig.org/wiki/display/JSG/cas-user


--
You are currently subscribed to cas-user@lists.jasig.org as: 
venu.a...@berkeley.edu
To unsubscribe, change settings or access archives, see 
http://www.ja-sig.org/wiki/display/JSG/cas-user


--
You are currently subscribed to cas-user@lists.jasig.org as: 
arch...@mail-archive.com
To unsubscribe, change settings or access archives, see 
http://www.ja-sig.org/wiki/display/JSG/cas-user