I have a couple Tomcat instances on a pair of servers and another pair
running the repcache patched version of memcached.
I thought that running the two memcache instances would be a little more
fault tolerant. But it seems when one of them goes down, Tomcat will
throw errors. I'm wondering if I have some setting misconfigured. The
errors will show in the window where someone is trying to get a ticket
for an application.
Below is from my catalina.log..
2009-01-09 01:12:46.000 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
#Rops=3, #Wops=0, #iq=0,
toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e,
topWop=null, toWrite=72, interested=5}
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:360)
at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:283)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:265)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
2009-01-09 01:12:46.002 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=3,
#Wops=0, #iq=0,
toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e,
topWop=null, toWrite=72, interested=5}, attempt 0.
2009-01-09 01:12:46.017 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op:
net.spy.memcached.protocol.ascii.storeoperationi...@128b30e
2009-01-09 01:12:46.017 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op:
net.spy.memcached.protocol.ascii.getoperationi...@16ba008
2009-01-09 01:12:46.018 WARN
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
partially completed op:
net.spy.memcached.protocol.ascii.getoperationi...@d78cb8
2009-01-09 01:12:46,055 ERROR
[org.apache.catalina.core.ContainerBase.[Catalina].[c-cas-02.dtcc.edu].[/cas].[cas]]
- <Servlet.service() for servlet cas threw exception>
java.lang.RuntimeException: Cancelled
at
net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1443)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)
at
org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:109)
at
org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFromInternal(TicketOrCredentialBasedAuditablePrincipalResolver.java:50)
at
org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFrom(TicketOrCredentialBasedAuditablePrincipalResolver.java:39)
at
org.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:116)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
at
org.inspektr.statistics.StatisticManagementAspect.handleStatisticGathering(StatisticManagementAspect.java:73)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy9.grantServiceTicket(Unknown Source)
at
org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)
at
org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)
at
org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)
at
org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)
at
org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
at org.springframework.webflow.engine.State.enter(State.java:191)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:212)
at
org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
at org.springframework.webflow.engine.State.enter(State.java:191)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:212)
at
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
at
org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
at org.springframework.webflow.engine.State.enter(State.java:191)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:212)
at
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
at
org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
at org.springframework.webflow.engine.State.enter(State.java:191)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:212)
at
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
at
org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
at org.springframework.webflow.engine.State.enter(State.java:191)
at
org.springframework.webflow.engine.Transition.execute(Transition.java:212)
at
org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
at
org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
at
org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)
at
org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)
at
org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)
at
org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:619)
2009-01-09 01:12:46.119 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
topRop=null, topWop=null, toWrite=0, interested=0}
2009-01-09 01:12:46.121 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.selectionkeyi...@b1625c
2009-01-09 01:12:46.123 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
#Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
2009-01-09 01:12:46.125 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=0,
#Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0},
attempt 1.
2009-01-09 01:12:46.328 INFO net.spy.memcached.MemcachedConnection:
Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
topRop=null, topWop=null, toWrite=0, interested=0}
2009-01-09 01:12:46.329 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.selectionkeyi...@16df0d6
2009-01-09 01:12:46.330 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
#Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
The four lines or so repeat themselves until I restart memcached. I can
grab some of my config files and whatnot, but I was curious what
experiences other people had with this.
Thanks in advance,
Pat
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Patrick Hennessy ([email protected])
Senior Systems Specialist
Division of Information and Educational Technology
Delaware Technical and Community College
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
_______________________________________________
Yale CAS mailing list
[email protected]
http://tp.its.yale.edu/mailman/listinfo/cas
