On configuring the following version of memcached/repcached
memcached-1.2.8-repcached-2.2.tar.gz 222.2 KiB Mon May 25 2009
19:18
I am getting the following error (This is when I am trying to run cas with
memcached registry on a single machine)
12:06:32,303 ERROR [STDERR] INFO
12:06:32,303 ERROR [STDERR]
12:06:32,303 ERROR [STDERR] net.spy.memcached.MemcachedConnection
12:06:32,303 ERROR [STDERR] :
12:06:32,303 ERROR [STDERR] Reconnecting due to exception on {QA
sa=/my.server.com:20700, #Rops=2, #Wops=0, #iq=0,
toprop=net.spy.memcached.protocol.ascii.storeoperationi...@869e8f,
topWop=null, toWrite=69, interested=5}
12:06:32,303 ERROR [STDERR] java.io.IOException: Broken pipe
12:06:32,304 ERROR [STDERR] at sun.nio.ch.FileDispatcher.write0(Native
Method)
12:06:32,304 ERROR [STDERR] at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
12:06:32,304 ERROR [STDERR] at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
12:06:32,304 ERROR [STDERR] at sun.nio.ch.IOUtil.write(IOUtil.java:75)
12:06:32,304 ERROR [STDERR] at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:302)
12:06:32,304 ERROR [STDERR] at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:360)
12:06:32,304 ERROR [STDERR] at
net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:283)
12:06:32,304 ERROR [STDERR] at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:265)
12:06:32,305 ERROR [STDERR] at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
12:06:32,305 ERROR [STDERR] at
net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
12:06:32,306 ERROR [STDERR] 2009-09-10 12:06:32.305
12:06:32,306 ERROR [STDERR]
12:06:32,306 ERROR [STDERR] WARN
12:06:32,306 ERROR [STDERR]
12:06:32,306 ERROR [STDERR] net.spy.memcached.MemcachedConnection
12:06:32,306 ERROR [STDERR] :
12:06:32,306 ERROR [STDERR] Closing, and reopening {QA
sa=/my.server.com:20700, #Rops=2, #Wops=0, #iq=0,
toprop=net.spy.memcached.protocol.ascii.storeoperationi...@869e8f,
topWop=null, toWrite=69, interested=5}, attempt 0.
12:06:32,309 ERROR [STDERR] 2009-09-10 12:06:32.309
12:06:32,309 ERROR [STDERR]
12:06:32,309 ERROR [STDERR] WARN
12:06:32,309 ERROR [STDERR]
12:06:32,309 ERROR [STDERR]
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl
12:06:32,309 ERROR [STDERR] :
12:06:32,309 ERROR [STDERR] Discarding partially completed op:
net.spy.memcached.protocol.ascii.storeoperationi...@869e8f
12:06:32,310 ERROR [STDERR] 2009-09-10 12:06:32.310
12:06:32,310 ERROR [STDERR]
12:06:32,310 ERROR [STDERR] WARN
12:06:32,310 ERROR [STDERR]
12:06:32,310 ERROR [STDERR]
net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl
12:06:32,310 ERROR [STDERR] :
12:06:32,310 ERROR [STDERR] Discarding partially completed op:
net.spy.memcached.protocol.ascii.getoperationi...@1f23ca4
12:06:32,415 ERROR [STDERR] 2009-09-10 12:06:32.415
12:06:32,415 ERROR [STDERR]
12:06:32,416 ERROR [STDERR] INFO
12:06:32,416 ERROR [STDERR]
12:06:32,416 ERROR [STDERR] net.spy.memcached.MemcachedConnection
12:06:32,416 ERROR [STDERR] :
12:06:32,416 ERROR [STDERR] Reconnecting {QA sa=/my.server.com:20700,
#Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
12:06:32,417 ERROR [STDERR] 2009-09-10 12:06:32.417
12:06:32,418 ERROR [STDERR]
12:06:32,418 ERROR [STDERR] INFO
12:06:32,418 ERROR [STDERR]
12:06:32,418 ERROR [STDERR] net.spy.memcached.MemcachedConnection
12:06:32,418 ERROR [STDERR] :
12:06:32,418 ERROR [STDERR] Connection state changed for
sun.nio.ch.selectionkeyi...@1479784
Any clues how to debug it? Where do I start to look for details of this
behavior?
Running this on jboss-4.2.3GA server on RHL : i686 athlon i386 GNU/Linux
I can see the memcached process running :
$ memcached -v -u myUser -d -m 512 -l 202.1.150.6 -p 11211
- Shivani.
scott_battaglia wrote:
>
> In general there may be a short window where it doesn't fail over
> correctly
> (the client basically checks the servers to see if the are up). If we're
> not correctly capturing that error and at least just returning null, we
> should.
> -Scott
>
> -Scott Battaglia
> PGP Public Key Id: 0x383733AA
> LinkedIn: http://www.linkedin.com/in/scottbattaglia
>
>
> On Fri, Jan 9, 2009 at 1:40 AM, Patrick Hennessy <[email protected]>
> wrote:
>
>>
>> I have a couple Tomcat instances on a pair of servers and another pair
>> running the repcache patched version of memcached.
>>
>> I thought that running the two memcache instances would be a little more
>> fault tolerant. But it seems when one of them goes down, Tomcat will
>> throw errors. I'm wondering if I have some setting misconfigured. The
>> errors will show in the window where someone is trying to get a ticket
>> for an application.
>>
>> Below is from my catalina.log..
>>
>>
>> 2009-01-09 01:12:46.000 INFO net.spy.memcached.MemcachedConnection:
>> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
>> #Rops=3, #Wops=0, #iq=0,
>> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e,
>> topWop=null, toWrite=72, interested=5}
>> java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcher.write0(Native Method)
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
>> at sun.nio.ch.IOUtil.write(IOUtil.java:75)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
>> at
>>
>> net.spy.memcached.protocol.TCPMemcachedNodeImpl.writeSome(TCPMemcachedNodeImpl.java:360)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleWrites(MemcachedConnection.java:283)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:265)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
>> at
>> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
>> 2009-01-09 01:12:46.002 WARN net.spy.memcached.MemcachedConnection:
>> Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=3,
>> #Wops=0, #iq=0,
>> toprop=net.spy.memcached.protocol.ascii.storeoperationi...@128b30e,
>> topWop=null, toWrite=72, interested=5}, attempt 0.
>> 2009-01-09 01:12:46.017 WARN
>> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
>> partially completed op:
>> net.spy.memcached.protocol.ascii.storeoperationi...@128b30e
>> 2009-01-09 01:12:46.017 WARN
>> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
>> partially completed op:
>> net.spy.memcached.protocol.ascii.getoperationi...@16ba008
>> 2009-01-09 01:12:46.018 WARN
>> net.spy.memcached.protocol.ascii.AsciiMemcachedNodeImpl: Discarding
>> partially completed op:
>> net.spy.memcached.protocol.ascii.getoperationi...@d78cb8
>> 2009-01-09 01:12:46,055 ERROR
>> [org.apache.catalina.core.ContainerBase.[Catalina].[c-cas-02.dtcc.edu
>> ].[/cas].[cas]]
>> - <Servlet.service() for servlet cas threw exception>
>> java.lang.RuntimeException: Cancelled
>> at
>>
>> net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1443)
>> at
>> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
>> at
>> net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)
>> at
>>
>> org.jasig.cas.ticket.registry.MemCacheTicketRegistry.getTicket(MemCacheTicketRegistry.java:109)
>> at
>>
>> org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFromInternal(TicketOrCredentialBasedAuditablePrincipalResolver.java:50)
>> at
>>
>> org.jasig.cas.audit.spi.TicketOrCredentialBasedAuditablePrincipalResolver.resolveFrom(TicketOrCredentialBasedAuditablePrincipalResolver.java:39)
>> at
>>
>> org.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:116)
>> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>>
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
>> at
>>
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
>> at
>>
>> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
>> at
>>
>> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:77)
>> at
>>
>> org.inspektr.statistics.StatisticManagementAspect.handleStatisticGathering(StatisticManagementAspect.java:73)
>> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>>
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
>> at
>>
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
>> at
>>
>> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:160)
>> at
>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
>> at
>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at $Proxy9.grantServiceTicket(Unknown Source)
>> at
>>
>> org.jasig.cas.web.flow.GenerateServiceTicketAction.doExecute(GenerateServiceTicketAction.java:37)
>> at
>>
>> org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:192)
>> at
>>
>> org.springframework.webflow.engine.AnnotatedAction.execute(AnnotatedAction.java:146)
>> at
>>
>> org.springframework.webflow.engine.ActionExecutor.execute(ActionExecutor.java:59)
>> at
>>
>> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:156)
>> at org.springframework.webflow.engine.State.enter(State.java:191)
>> at
>> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
>> at
>>
>> org.springframework.webflow.engine.DecisionState.doEnter(DecisionState.java:54)
>> at org.springframework.webflow.engine.State.enter(State.java:191)
>> at
>> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
>> at
>>
>> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
>> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
>> at
>>
>> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
>> at
>>
>> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
>> at org.springframework.webflow.engine.State.enter(State.java:191)
>> at
>> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
>> at
>>
>> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
>> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
>> at
>>
>> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
>> at
>>
>> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
>> at org.springframework.webflow.engine.State.enter(State.java:191)
>> at
>> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
>> at
>>
>> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
>> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
>> at
>>
>> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
>> at
>>
>> org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:161)
>> at org.springframework.webflow.engine.State.enter(State.java:191)
>> at
>> org.springframework.webflow.engine.Transition.execute(Transition.java:212)
>> at
>>
>> org.springframework.webflow.engine.TransitionableState.onEvent(TransitionableState.java:107)
>> at org.springframework.webflow.engine.Flow.onEvent(Flow.java:534)
>> at
>>
>> org.springframework.webflow.engine.impl.RequestControlContextImpl.signalEvent(RequestControlContextImpl.java:205)
>> at
>>
>> org.springframework.webflow.engine.impl.FlowExecutionImpl.signalEvent(FlowExecutionImpl.java:202)
>> at
>>
>> org.springframework.webflow.executor.FlowExecutorImpl.resume(FlowExecutorImpl.java:222)
>> at
>>
>> org.springframework.webflow.executor.support.FlowRequestHandler.handleFlowRequest(FlowRequestHandler.java:111)
>> at
>>
>> org.springframework.webflow.executor.mvc.FlowController.handleRequestInternal(FlowController.java:165)
>> at
>>
>> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
>> at
>>
>> org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
>> at
>>
>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
>> at
>>
>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
>> at
>>
>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
>> at
>>
>> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>> at
>>
>> org.jasig.cas.web.init.SafeDispatcherServlet.service(SafeDispatcherServlet.java:115)
>> at
>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>> at
>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>>
>> org.inspektr.common.web.ClientInfoThreadLocalFilter.doFilterInternal(ClientInfoThreadLocalFilter.java:48)
>> at
>>
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
>> at
>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>> at
>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>>
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
>> at
>>
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>> at
>>
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
>> at
>>
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>>
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>> at
>>
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
>> at
>>
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>> at
>>
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>> at
>>
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>> at
>>
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
>> at java.lang.Thread.run(Thread.java:619)
>> 2009-01-09 01:12:46.119 INFO net.spy.memcached.MemcachedConnection:
>> Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
>> topRop=null, topWop=null, toWrite=0, interested=0}
>> 2009-01-09 01:12:46.121 INFO net.spy.memcached.MemcachedConnection:
>> Connection state changed for sun.nio.ch.selectionkeyi...@b1625c
>> 2009-01-09 01:12:46.123 INFO net.spy.memcached.MemcachedConnection:
>> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
>> #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0,
>> interested=0}
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
>> at
>> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
>> 2009-01-09 01:12:46.125 WARN net.spy.memcached.MemcachedConnection:
>> Closing, and reopening {QA sa=hostname/ip_address:11211, #Rops=0,
>> #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0},
>> attempt 1.
>> 2009-01-09 01:12:46.328 INFO net.spy.memcached.MemcachedConnection:
>> Reconnecting {QA sa=hostname/ip_address:11211, #Rops=0, #Wops=0, #iq=0,
>> topRop=null, topWop=null, toWrite=0, interested=0}
>> 2009-01-09 01:12:46.329 INFO net.spy.memcached.MemcachedConnection:
>> Connection state changed for sun.nio.ch.selectionkeyi...@16df0d6
>> 2009-01-09 01:12:46.330 INFO net.spy.memcached.MemcachedConnection:
>> Reconnecting due to exception on {QA sa=hostname/ip_address:11211,
>> #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0,
>> interested=0}
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250)
>> at
>>
>> net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:182)
>> at
>> net.spy.memcached.MemcachedClient.run(MemcachedClient.java:1259)
>>
>>
>> The four lines or so repeat themselves until I restart memcached. I can
>> grab some of my config files and whatnot, but I was curious what
>> experiences other people had with this.
>>
>> Thanks in advance,
>>
>> Pat
>>
>> --
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>> Patrick Hennessy ([email protected])
>>
>> Senior Systems Specialist
>> Division of Information and Educational Technology
>> Delaware Technical and Community College
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>
>> _______________________________________________
>> Yale CAS mailing list
>> [email protected]
>> http://tp.its.yale.edu/mailman/listinfo/cas
>>
>
> _______________________________________________
> Yale CAS mailing list
> [email protected]
> http://tp.its.yale.edu/mailman/listinfo/cas
>
>
--
View this message in context:
http://www.nabble.com/MemCacheTicketRegistry-and-memcached-repcached-failure-tp21367068p25378198.html
Sent from the CAS Users mailing list archive at Nabble.com.
--
You are currently subscribed to [email protected] as:
[email protected]
To unsubscribe, change settings or access archives, see
http://www.ja-sig.org/wiki/display/JSG/cas-user
