[
https://issues.apache.org/jira/browse/CASSANDRA-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Overton reopened CASSANDRA-3677:
------------------------------------
I've been able to reproduce this in 1.07 and in trunk as follows:
* create a cluster of 2 nodes
* stop one of the nodes
* insert some data at RF=1, CL=ANY - this causes hints to be stored
* nodetool removetoken on the token of the dead node
* some time up to 10 minutes later the exception is logged:
ERROR [OptionalTasks:1] 2012-02-07 14:41:57,710 AbstractCassandraDaemon.java
(line 134) Fatal exception in thread Thread[OptionalTasks:1,5,main]
java.lang.NullPointerException
at
org.cliffc.high_scale_lib.NonBlockingHashMap.hash(NonBlockingHashMap.java:113)
at
org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:553)
at
org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:348)
at
org.cliffc.high_scale_lib.NonBlockingHashMap.putIfAbsent(NonBlockingHashMap.java:319)
at
org.cliffc.high_scale_lib.NonBlockingHashSet.add(NonBlockingHashSet.java:32)
at
org.apache.cassandra.db.HintedHandOffManager.scheduleHintDelivery(HintedHandOffManager.java:410)
at
org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:395)
at
org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:84)
at
org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:119)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Adverse Effects:
* The 10min repeating scheduleAllDeliveries() introduced in CASSANDRA-3554 will
no longer fire (since the exception was uncaught), so the node has now
regressed to pre-CASSANDRA-3554 behaviour
* Even after the node is restarted, the exception will be thrown again next
time the schedule is run.
* This will continue until the row tombstone for the dead node's hints is
garbage-collected (10 days)
Fix:
* don't try to schedule delivery of hints for tokens which are no longer valid
endpoints (see attached).
> NPE during HH delivery when gossip turned off on target
> -------------------------------------------------------
>
> Key: CASSANDRA-3677
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3677
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.7
> Reporter: Radim Kolar
> Assignee: Brandon Williams
> Priority: Trivial
> Attachments: 3677.txt
>
>
> probably not important bug
> ERROR [OptionalTasks:1] 2011-12-27 21:44:25,342 AbstractCassandraDaemon.java
> (line 138) Fatal exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.NullPointerException
> at
> org.cliffc.high_scale_lib.NonBlockingHashMap.hash(NonBlockingHashMap.java:113)
> at
> org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:553)
> at
> org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:348)
> at
> org.cliffc.high_scale_lib.NonBlockingHashMap.putIfAbsent(NonBlockingHashMap.java:319)
> at
> org.cliffc.high_scale_lib.NonBlockingHashSet.add(NonBlockingHashSet.java:32)
> at
> org.apache.cassandra.db.HintedHandOffManager.scheduleHintDelivery(HintedHandOffManager.java:371)
> at
> org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:356)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:84)
> at
> org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:119)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
