I don't know the memcached server off the top of my head but the CAS version should be like 3.3.2 or something. We're running it on Solaris. Again, our memcache interaction is not going through any form of load balancer or device.
Cheers, Scott On Fri, Jul 24, 2009 at 1:07 PM, venu.alla <[email protected]> wrote: > Scott, > > What versions of CAS and Memcache+Repcached are your running and what is > the OS? > > Here is what we are running > CAS: 3.3.1 > memcached-1.2.8-repcached-2.2.tar.gz > > Venu > > Scott Battaglia wrote: > >> We're essentially running >> >> Load Balancer -> CAS (x2) Each of the CAS machines has a memcache >> instance on it and CAS refers to the internal name of the memache server (it >> does not go through the load balancer). >> >> >> >> On Thu, Jul 23, 2009 at 2:24 PM, venu.alla <[email protected]<mailto: >> [email protected]>> wrote: >> >> Scott, >> >> We ran into a boad loat of "Connection timed out" issues starting >> last night and we had to restart cas server to return to normal >> state. CAS was essentially unfunctional during this state. >> >> We are hosting cas and memcache on its own server, OS is RH. >> >> Our deployment is as follows. >> >> cas ---> LB ----> memcache[1,2] with repcache. >> >> What is your env and deployment like? >> Venu >> >> Here is the stack trace we saw upon login attempt..... >> >> >> *message* >> >> *description* The server encountered an internal error () that >> prevented it from fulfilling this request. >> >> *exception* >> >> org.springframework.web.util.NestedServletException: Request >> processing failed; nested exception is >> org.springframework.webflow.engine.ActionExecutionException: >> Exception thrown executing [annotatedact...@67e90fee targetAction = >> org.jasig.cas.web.flow.generateserviceticketact...@7615078d, >> attributes = map[[empty]]] in state 'generateServiceTicket' of flow >> 'login-webflow' -- action execution attributes were 'map[[empty]]'; >> nested exception is net.spy.memcached.OperationTimeoutException: >> Timeout waiting for value >> >> >> >> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:583) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:637) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >> >> >> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) >> >> >> >> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) >> >> >> >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) >> >> *root cause* >> >> org.springframework.webflow.engine.ActionExecutionException: >> Exception thrown executing [annotatedact...@67e90fee targetAction = >> org.jasig.cas.web.flow.generateserviceticketact...@7615078d, >> attributes = map[[empty]]] in state 'generateServiceTicket' of flow >> 'login-webflow' -- action execution attributes were 'map[[empty]]'; >> nested exception is net.spy.memcached.OperationTimeoutException: >> Timeout waiting for value >> >> >> >> org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:64) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> >> org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) >> >> >> >> org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) >> >> >> >> org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) >> >> >> >> org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) >> >> >> >> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) >> >> >> >> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:637) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >> >> >> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) >> >> >> >> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) >> >> >> >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) >> >> *root cause* >> >> net.spy.memcached.OperationTimeoutException: Timeout waiting for value >> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:771) >> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784) >> >> >> >> org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119) >> >> >> >> org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35) >> >> >> >> org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173) >> >> >> >> org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245) >> >> >> >> org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37) >> >> >> >> org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192) >> >> >> >> org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146) >> >> >> >> org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> >> org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) >> >> >> >> org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) >> >> >> >> org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) >> >> >> >> org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) >> >> >> >> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) >> >> >> >> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:637) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >> >> >> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) >> >> >> >> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) >> >> >> >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) >> >> *root cause* >> >> java.util.concurrent.TimeoutException: Timed out waiting for operation >> >> >> >> net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1437) >> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764) >> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784) >> >> >> >> org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:119) >> >> >> >> org.jasig.cas.ticket.registry.AbstractTicketRegistry.getTicket(AbstractTicketRegistry.java:35) >> >> >> >> org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:173) >> >> >> >> org.jasig.cas.CentralAuthenticationServiceImpl.grantServiceTicket(CentralAuthenticationServiceImpl.java:245) >> >> >> >> org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37) >> >> >> >> org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192) >> >> >> >> org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146) >> >> >> >> org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) >> org.springframework.webflow.engine.State.enter(State.java:191) >> >> >> org.springframework.webflow.engine.Transition.execute(Transition.java:212) >> >> >> >> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) >> org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) >> >> >> >> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) >> >> >> >> org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) >> >> >> >> org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) >> >> >> >> org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) >> >> >> >> org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) >> >> >> >> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) >> >> >> >> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) >> >> >> >> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) >> >> >> >> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:637) >> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >> >> >> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) >> >> >> >> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) >> >> >> >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) >> >> *note* The full stack trace of the root cause is available in the >> Apache Tomcat/6.0.18 logs. >> >> >> >> >> Scott Battaglia wrote: >> >> I just checked our logs and we've only encountered connection >> reset by peer 10 times since 11/14/2008. Is there any specific >> setting you want me to check for you? >> >> Cheers, >> Scott >> >> >> On Mon, Jul 20, 2009 at 1:36 PM, venu.alla >> <[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> >> wrote: >> >> hi, >> We are currently testing Memcached(with repcached) as CAS >> Service >> Ticket Registry. We see the following errors intermittently. >> Any one >> else seen these in their own env and if so, any feed back as >> to why >> and how to reduce them. We also notice that, when timedout error >> occurs, the exception is thrown all the way up into tomcat >> which in >> turn shows up on end user page. >> >> 1) Connection Timed out (rare but impacts service availability) >> 2) Connection Reset by Peer ( more frequently than type-1) >> >>>>>>>TIMED OUT>>>>>>>>>>>>>>> >> 2009-07-20 08:26:50.385 INFO >> net.spy.memcached.MemcachedConnection: >> Reconnecting due to exception on {QA >> sa=xxxx.berkeley.edu/x.xx.xx.xx:11211 >> <http://xxxx.berkeley.edu/x.xx.xx.xx:11211> >> <http://xxxx.berkeley.edu/x.xx.xx.xx:11211>, #Rops=28, #Wops=0, >> >> #iq=0, >> >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77, >> topWop=null, toWrite=0, interested=1} >> java.io.IOException: Connection timed out >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at >> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at >> sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:206) >> at >> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> at >> >> net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) >> at >> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) >> 2009-07-20 08:26:50.386 WARN >> net.spy.memcached.MemcachedConnection: >> Closing, and reopening {QA sa=xxxx.berkeley.edu/x.x.x.x:11211 >> <http://xxxx.berkeley.edu/x.x.x.x:11211> >> <http://xxxx.berkeley.edu/x.x.x.x:11211>, #Rops=28, #Wops=0, >> #iq=0, >> >> >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@766eae77, >> topWop=null, toWrite=0, interested=1}, attempt 0. >> 2009-07-20 08:26:50.387 WARN >> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: >> Discarding >> partially completed op: >> Note: Many discards after this before it returned to normal >> state. >> >> >>>>>>>>>>>> Connection RESET >>>>>>>>>>>>>>>>>>>>>>>> >> 2009-07-20 00:38:01.390 INFO >> net.spy.memcached.MemcachedConnection: >> Reconnecting due to exception on {QA >> sa=xx.x.edu/x.x.x.x:11211 <http://xx.x.edu/x.x.x.x:11211> >> <http://xx.x.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, #iq=0, >> >> >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca, >> topWop=null, toWrite=0, interested=1} >> java.io.IOException: Connection reset by peer >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at >> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at >> sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:206) >> at >> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> at >> >> net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:294) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) >> at >> >> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) >> at >> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) >> 2009-07-20 00:38:01.390 WARN >> net.spy.memcached.MemcachedConnection: >> Closing, and reopening {QA sa=xxx.berkeley.edu/x.x.x.x:11211 >> <http://xxx.berkeley.edu/x.x.x.x:11211> >> <http://xxx.berkeley.edu/x.x.x.x:11211>, #Rops=2, #Wops=0, >> #iq=0, >> >> >> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@fa0a4ca, >> topWop=null, toWrite=0, interested=1}, attempt 0. >> >> >> -- You are currently subscribed to >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>> as: [email protected] >> <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>> >> >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> >> -- You are currently subscribed to [email protected] >> <mailto:[email protected]> as: [email protected] >> <mailto:[email protected]> >> >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> >> -- You are currently subscribed to [email protected] >> <mailto:[email protected]> as: [email protected] >> <mailto:[email protected]> >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> >> -- >> You are currently subscribed to [email protected] as: >> [email protected] >> To unsubscribe, change settings or access archives, see >> http://www.ja-sig.org/wiki/display/JSG/cas-user >> >> > -- > You are currently subscribed to [email protected] as: > [email protected] > To unsubscribe, change settings or access archives, see > http://www.ja-sig.org/wiki/display/JSG/cas-user > -- You are currently subscribed to [email protected] as: [email protected] To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user
