In general there may be a short window where it doesn't fail over correctly (the client basically checks the servers to see if the are up). If we're not correctly capturing that error and at least just returning null, we should. -Scott
-Scott Battaglia PGP Public Key Id: 0x383733AA LinkedIn: http://www.linkedin.com/in/scottbattaglia On Fri, Jan 9, 2009 at 1:40 AM, Patrick Hennessy <[email protected]> wrote: > > I have a couple Tomcat instances on a pair of servers and another pair > running the repcache patched version of memcached. > > I thought that running the two memcache instances would be a little more > fault tolerant. But it seems when one of them goes down, Tomcat will > throw errors. I'm wondering if I have some setting misconfigured. The > errors will show in the window where someone is trying to get a ticket > for an application. > > Below is from my catalina.log.. > > > 2009-01-09 01:12:46.000 INFO net.spy.memcached.MemcachedConnection: > Reconnecting due to exception on {QA sa=hostname/ip_address:11211, > #Rops=3, #Wops=0, #iq=0, > toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e, > topWop=null, toWrite=72, interested=5} > java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcher.write0(Native Method) > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) > at sun.nio.ch.IOUtil.write(IOUtil.java:75) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) > at > > net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:360) > at > > net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:283) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:265) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) > at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) > 2009-01-09 01:12:46.002 WARN net.spy.memcached.MemcachedConnection: > Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=3, > #Wops=0, #iq=0, > toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e, > topWop=null, toWrite=72, interested=5}, attempt 0. > 2009-01-09 01:12:46.017 WARN > net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding > partially completed op: > net.spy.memcached.protocol.ascii.storeoperationi...@128b30e > 2009-01-09 01:12:46.017 WARN > net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding > partially completed op: > net.spy.memcached.protocol.ascii.getoperationi...@16ba008 > 2009-01-09 01:12:46.018 WARN > net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding > partially completed op: > net.spy.memcached.protocol.ascii.getoperationi...@d78cb8 > 2009-01-09 01:12:46,055 ERROR > [org.apache.catalina.core.ContainerBase.[Catalina].[c-cas-02.dtcc.edu > ].[/cas].[cas]] > - <Servlet.service() for servlet cas threw exception> > java.lang.RuntimeException: Cancelled > at > > net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1443) > at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764) > at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784) > at > > org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:109) > at > > org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFromInternal(TicketOrCredentialBasedAuditablePrincipalResolver.java:50) > at > > org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFrom(TicketOrCredentialBasedAuditablePrincipalResolver.java:39) > at > > org.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:116) > at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) > at > > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) > at > > org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64) > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160) > at > > org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77) > at > > org.inspektr.statistics.StatisticManagementAspect.handleStatisticGathering(StatisticManagementAspect.java:73) > at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) > at > > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) > at > > org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64) > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160) > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy9.grantServiceTicket(Unknown Source) > at > > org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37) > at > > org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192) > at > > org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146) > at > > org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59) > at > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156) > at org.springframework.webflow.engine.State.enter(State.java:191) > at > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > at > > org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54) > at org.springframework.webflow.engine.State.enter(State.java:191) > at > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > at > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > at > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > at > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > at org.springframework.webflow.engine.State.enter(State.java:191) > at > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > at > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > at > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > at > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > at org.springframework.webflow.engine.State.enter(State.java:191) > at > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > at > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > at > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > at > > org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161) > at org.springframework.webflow.engine.State.enter(State.java:191) > at > org.springframework.webflow.engine.Transition.execute(Transition.java:212) > at > > org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107) > at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534) > at > > org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205) > at > > org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202) > at > > org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222) > at > > org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111) > at > > org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165) > at > > org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) > at > > org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) > at > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) > at > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) > at > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) > at > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > at > > org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > > org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48) > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870) > at > > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) > at java.lang.Thread.run(Thread.java:619) > 2009-01-09 01:12:46.119 INFO net.spy.memcached.MemcachedConnection: > Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0, > topRop=null, topWop=null, toWrite=0, interested=0} > 2009-01-09 01:12:46.121 INFO net.spy.memcached.MemcachedConnection: > Connection state changed for sun.nio.ch.selectionkeyi...@b1625c > 2009-01-09 01:12:46.123 INFO net.spy.memcached.MemcachedConnection: > Reconnecting due to exception on {QA sa=hostname/ip_address:11211, > #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) > at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) > 2009-01-09 01:12:46.125 WARN net.spy.memcached.MemcachedConnection: > Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=0, > #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, > attempt 1. > 2009-01-09 01:12:46.328 INFO net.spy.memcached.MemcachedConnection: > Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0, > topRop=null, topWop=null, toWrite=0, interested=0} > 2009-01-09 01:12:46.329 INFO net.spy.memcached.MemcachedConnection: > Connection state changed for sun.nio.ch.selectionkeyi...@16df0d6 > 2009-01-09 01:12:46.330 INFO net.spy.memcached.MemcachedConnection: > Reconnecting due to exception on {QA sa=hostname/ip_address:11211, > #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250) > at > > net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182) > at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259) > > > The four lines or so repeat themselves until I restart memcached. I can > grab some of my config files and whatnot, but I was curious what > experiences other people had with this. > > Thanks in advance, > > Pat > > -- > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Patrick Hennessy ([email protected]) > > Senior Systems Specialist > Division of Information and Educational Technology > Delaware Technical and Community College > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > _______________________________________________ > Yale CAS mailing list > [email protected] > http://tp.its.yale.edu/mailman/listinfo/cas >
_______________________________________________ Yale CAS mailing list [email protected] http://tp.its.yale.edu/mailman/listinfo/cas
