[
https://issues.apache.org/jira/browse/CASSANDRA-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940540#comment-14940540
]
Andrew Hust commented on CASSANDRA-10233:
-----------------------------------------
[~eitikimura] as [~pauloricardomg] mentioned truncating the hints table will
get things rolling again, just make sure to repair in near future to prevent
possible dataloss or zombie records related to the mutations truncated out of
the hints table.
[~fhsgoncalves] thanks, that helps a ton. Couple more questions to help
reproduce it:
- Just want to make sure order of operation is correct, the error showed up
when you added additional nodes then persisted after upgrades?
- Are all the new nodes in the second rack?
- Are you running with vnodes?
- Are your keyspaces set with a replication of NetworkTopologyStrategy,
SimpleStrategy or ?
- In the cassandra.yaml what is the value for: endpoint_snitch
> IndexOutOfBoundsException in HintedHandOffManager
> -------------------------------------------------
>
> Key: CASSANDRA-10233
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10233
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.0
> Reporter: Omri Iluz
> Assignee: Andrew Hust
> Attachments: cassandra-2.1.8-10233-v2.txt, cassandra-2.1.8-10233.txt
>
>
> After upgrading our cluster to 2.2.0, the following error started showing
> exectly every 10 minutes on every server in the cluster:
> {noformat}
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,506
> CompactionTask.java:142 - Compacting (8e7e1520-500e-11e5-b1e3-e95897ba4d20)
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-540-big-Data.db:level=0,
> ]
> INFO [CompactionExecutor:1381] 2015-08-31 18:31:55,599
> CompactionTask.java:224 - Compacted (8e7e1520-500e-11e5-b1e3-e95897ba4d20) 1
> sstables to
> [/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-541-big,]
> to level=0. 1,544,495 bytes to 1,544,495 (~100% of original) in 93ms =
> 15.838121MB/s. 0 total partitions merged to 4. Partition merge counts were
> {1:4, }
> ERROR [HintedHandoff:1] 2015-08-31 18:31:55,600 CassandraDaemon.java:182 -
> Exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.IndexOutOfBoundsException: null
> at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_79]
> at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:410)
> ~[na:1.7.0_79]
> at org.apache.cassandra.utils.UUIDGen.getUUID(UUIDGen.java:106)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:515)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:88)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:168)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_79]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_79]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
