I *think* that code is used when one node tells others via gossip it is
removing a token that is not it's own. The ode that receives information in
gossip does some work and then replies to the first node with a
REPLICATION_FINISHED message, which is the node I assume the error is happening
on.
Have you been doing any moves / removes or additions or tokens/nodes?
Thanks
Aaron
On 28 Apr 2011, at 08:39, Alexis Lê-Quôc wrote:
> Hi,
>
> I've been getting the following lately, every few seconds.
>
> 2011-04-27T20:21:18.299885+00:00 10.202.61.193 [MiscStage: 97] Error
> in ThreadPoolExecutor
> 2011-04-27T20:21:18.299885+00:00 10.202.61.193 java.lang.AssertionError
> 2011-04-27T20:21:18.300038+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.service.StorageService.confirmReplication(StorageService.java:1872)
> 2011-04-27T20:21:18.300038+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.streaming.ReplicationFinishedVerbHandler.doVerb(ReplicationFinishedVerbHandler.java:38)
> 2011-04-27T20:21:18.300047+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> 2011-04-27T20:21:18.300047+00:00 10.202.61.193 10.202.61.193 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 2011-04-27T20:21:18.300055+00:00 10.202.61.193 10.202.61.193 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 2011-04-27T20:21:18.300055+00:00 10.202.61.193 10.202.61.193 at
> java.lang.Thread.run(Thread.java:636)
> 2011-04-27T20:21:18.300555+00:00 10.202.61.193 [MiscStage: 97] Fatal
> exception in thread Thread[MiscStage:97,5,main]
>
> I see it coming from
> 32 public class ReplicationFinishedVerbHandler implements IVerbHandler
> 33 {
> 34 private static Logger logger =
> LoggerFactory.getLogger(ReplicationFinishedVerbHandler.class);
> 35
> 36 public void doVerb(Message msg, String id)
> 37 {
> 38 StorageService.instance.confirmReplication(msg.getFrom());
> 39 Message response =
> msg.getInternalReply(ArrayUtils.EMPTY_BYTE_ARRAY);
> 40 if (logger.isDebugEnabled())
> 41 logger.debug("Replying to " + id + "@" + msg.getFrom());
> 42 MessagingService.instance().sendReply(response, id, msg.getFrom());
> 43 }
> 44 }
>
> Before I dig deeper in the code, has anybody dealt with this before?
>
> Thanks,
>
> --
> Alexis Lê-Quôc
