java.util.NoSuchElementException when returning a node to the cluster
---------------------------------------------------------------------
Key: CASSANDRA-1432
URL: https://issues.apache.org/jira/browse/CASSANDRA-1432
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7 beta 1
Reporter: Aaron Morton
Priority: Minor
I'm running the v0.7-beta1 in a 4 nodes cluster and just doing some simple
testing. One of the nodes had been down (machine off, unclean shutdown) for an
hour or so not sure how many writes were going on, when I bought it back up
this message appears in the other 3 nodes...
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 584) Node
/192.168.34.27 has restarted, now UP again
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,200 HintedHandOffManager.java
(line 191) Started hinted handoff for endpoint /192.168.34.27
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,201 StorageService.java (line 636)
Node /192.168.34.27 state jump to normal
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,201 StorageService.java (line 643)
Will not change my token ownership to /192.168.34.27
ERROR [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,640 CassandraDaemon.java
(line 82) Uncaught exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main]
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.util.NoSuchElementException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.util.NoSuchElementException
at
orgapache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
... 2 more
Caused by: java.util.NoSuchElementException
at
java.util.concurrent.ConcurrentSkipListMap.lastKey(ConcurrentSkipListMap.java:1981)
at
java.util.concurrent.ConcurrentSkipListMap$KeySet.last(ConcurrentSkipListMap.java:2331)
at
org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:121)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:218)
at
org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:78)
at
org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:296)
not sure how many writes were going on
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
On the machine that was off (34.27) there are no errors in the logs, and here
are the entries for around the same time...
INFO [main] 2010-08-25 19:29:50,679 CommitLog.java (line 340) Recovery complete
INFO [main] 2010-08-25 19:29:50,769 CommitLog.java (line 180) Log replay
complete
INFO [main] 2010-08-25 19:29:50,797 StorageService.java (line 342) Cassandra
version: 0.7.0-beta1-SNAPSHOT
INFO [main] 2010-08-25 19:29:50,797 StorageService.java (line 343) Thrift API
version: 10.0.0
INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 240) Saved Token
found: 85070591730234615865843651857942052864
INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 257) Saved
ClusterName found: FOO
INFO [main] 2010-08-25 19:29:50,813 SystemTable.java (line 272) Saved
partitioner not found. Using org.apache.cassandra.dht.RandomPartitioner
INFO [main] 2010-08-25 19:29:50,814 ColumnFamilyStore.java (line 422)
switching in a fresh Memtable for LocationInfo at
CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-12827213897
70.log', position=41336)
INFO [main] 2010-08-25 19:29:50,814 ColumnFamilyStore.java (line 706)
Enqueuing flush of memtable-locationi...@916236367(95 bytes, 2 operations)
INFO [FLUSH-WRITER-POOL:1] 2010-08-25 19:29:50,815 Memtable.java (line 150)
Writing memtable-locationi...@916236367(95 bytes, 2 operations)
INFO [FLUSH-WRITER-POOL:1] 2010-08-25 19:29:50,873 Memtable.java (line 157)
Completed flushing
/local1/junkbox/cassandra/data/system/LocationInfo-e-6-Data.db
INFO [main] 2010-08-25 19:29:50,917 StorageService.java (line 374) Starting up
server gossip
INFO [main] 2010-08-25 19:29:51,093 ColumnFamilyStore.java (line 1239) Loaded
0 rows into the Super2 cache
INFO [main] 2010-08-25 19:29:51,170 CassandraDaemon.java (line 153) Binding
thrift service to /0.0.0.0:9160
INFO [main] 2010-08-25 19:29:51,174 CassandraDaemon.java (line 167) Using
TFramedTransport with a max frame size of 15728640 bytes.
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,198 Gossiper.java (line 578) Node
/192.168.34.28 is now part of the cluster
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 578) Node
/192.168.34.29 is now part of the cluster
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,199 Gossiper.java (line 578) Node
/192.168.34.26 is now part of the cluster
INFO [main] 2010-08-25 19:29:51,204 CassandraDaemon.java (line 208) Listening
for thrift clients...
INFO [main] 2010-08-25 19:29:51,210 Mx4jTool.java (line 73) Will not load
MX4J, mx4j-tools.jar is not in the classpath
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,417 HintedHandOffManager.java
(line 191) Started hinted handoff for endpoint /192.168.34.28
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,417 Gossiper.java (line 570)
InetAddress /192.168.34.28 is now UP
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,418 HintedHandOffManager.java
(line 247) Finished hinted handoff of 0 rows to endpoint /192.168.34.28
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,855 HintedHandOffManager.java
(line 191) Started hinted handoff for endpoint /192.168.34.29
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:51,855 Gossiper.java (line 570)
InetAddress /192.168.34.29 is now UP
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:51,860 HintedHandOffManager.java
(line 247) Finished hinted handoff of 0 rows to endpoint /192.168.34.29
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:52,930 HintedHandOffManager.java
(line 191) Started hinted handoff for endpoint /192.168.34.26
INFO [GOSSIP_STAGE:1] 2010-08-25 19:29:52,930 Gossiper.java (line 570)
InetAddress /192.168.34.26 is now UP
INFO [HINTED-HANDOFF-POOL:1] 2010-08-25 19:29:52,930 HintedHandOffManager.java
(line 247) Finished hinted handoff of 0 rows to endpoint /192.168.34.26
I ran a repair on all the nodes and this was all that they each logged
INFO [manual-repair-fe7c5abb-bb0a-4415-aa75-0d72ba4e7f1b] 2010-08-25
19:49:24,194 AntiEntropyService.java (line 803) Waiting for repair requests to:
[]
The cluster seemed OK and kept on working.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
