Faulty memory causes adjacent nodes to have invalid data and fail compactation
(java.io.IOException: Keys must be written in ascending order.)
----------------------------------------------------------------------------------------------------------------------------------------------
Key: CASSANDRA-2408
URL: https://issues.apache.org/jira/browse/CASSANDRA-2408
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.4
Reporter: Thibaut
Fix For: 0.7.5
Hi,
We had to replace a node with faulty ram. Besides the cassandra cluster getting
unresponsive, we also observerd a "keys must be written in ascending order"
exception on all adjacent quorum nodes, causing the compactation to fail.
We had another node with faulty ram a week earlier, and it was causing the same
errors on it's neighbours.
A faulty node shouldn't affect the key ordering of adjacent nodes?
INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java
(line 396) Compacting
[SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')]
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line
108) Last written key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
109) Current key :
DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4,
62613537386265643565373561303664313135366461626363633638616264355f7777772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f393232333337303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534)
INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line
110) Writing into file
/cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db
ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516
AbstractCassandraDaemon.java (line 112) Fatal exception in thread
Thread[CompactionExecutor:1,1,main]
java.io.IOException: Keys must be written in ascending order.
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
