Ok i've run scrub on the 3 nodes and the problematic row
Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
The full message is
WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041
CompactionManager.java (line 582) Non-fatal error reading row
(stacktrace follows)
java.io.IOError: java.io.IOException: Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:97)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:143)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:566)
at
org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:473)
at
org.apache.cassandra.db.compaction.CompactionManager.access$200(CompactionManager.java:64)
at
org.apache.cassandra.db.compaction.CompactionManager$3.perform(CompactionManager.java:213)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:241)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110)
... 13 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: Not enough
bytes to read value of component 1
at
org.apache.cassandra.db.marshal.AbstractCompositeType.validate(AbstractCompositeType.java:240)
at org.apache.cassandra.db.Column.validateName(Column.java:273)
at org.apache.cassandra.db.Column.validateFields(Column.java:278)
at
org.apache.cassandra.db.ColumnFamily.validateColumnFields(ColumnFamily.java:372)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:237)
... 14 more
WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,085
CompactionManager.java (line 624) Row at 4047368880 is unreadable;
skipping to next
This happened on several sstables on on each of the nodes - meaning it
was mutated several times
dsc2b:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-450-Data.db at
4244390041
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-452-Data.db at
9366462649
dsc2c:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-413-Data.db at
4047368880
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-481-Data.db at
3598063925
dsc1a:
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-883-Data.db at
271195463
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-733-Data.db at
1089815977
/var/lib/cassandra/data/PRODUCTION/UserCompletions-hc-895-Data.db at
312147765
How do I use this info to find out what is actually wrong with this
column, as I highly want to prevent it from being written again :)
Thanks
On 06/14/2012 05:02 PM, Sylvain Lebresne wrote:
Is there way to make cassandra throw away the offending column?
Running scrub should allow to get read of the row containing the
problematic column. Unfortunately it will discard the whole row, not
just the column.
However, since scrub takes a snapshot anyway (and should tell you
which sstable had the offending column), you could do that and keep
the bad sstable around to further investigate.
maybe sstable2json can somehow help?
It's worth a try but there is a good change sstable2json will throw an
error too. It may have written enough at that point for you to find
which row is wrong though.
--
Sylvain
