Hi,
I have a pretty urgent issue with 1.0.9 cluster
in opscenter i saw a compation that had a progress of 0% for a long
time, looking at the cassandra log on the relevant node I see REPEATED
messages of IllegalArgumentException in CompactionExecutor
INFO [CompactionExecutor:3335]
On Thu, Jun 14, 2012 at 8:26 AM, Piavlo lolitus...@gmail.com wrote:
I started looking for similar messages on other nodes saw a SINGLE
IllegalArgumentException on
ValidationExecutor on the same node and 2 other nodes (this is a 6 node
cluster) which happened
at almost the same time , in all
Hi Sylvain,
Yes this UserCompletions CF uses composite comparator and I do use
sstable compression.
What's the procedure to check if the compressed sstable is corrupted or not?
If it's corrupted what can I do to fix the issue with minimal cluster
load impact?
Is there way to delete all
On Thu, Jun 14, 2012 at 12:00 PM, Piavlo lolitus...@gmail.com wrote:
What's the procedure to check if the compressed sstable is corrupted or not?
Since you use compression, in theory that can't be disk bitrot since
in that case
you would have got some checksum error instead. The fact that it
Hi again,
After some further investigation now I'm in a situation that there are
3 nodes (of 6 nodes cluster) and all of them are falling with
ValidationExecutor during compaction which is is trigger by
repair -pr PRODUCTION UserCompletions against any node in the cluster
- repair get's
Is there way to make cassandra throw away the offending column?
Running scrub should allow to get read of the row containing the
problematic column. Unfortunately it will discard the whole row, not
just the column.
However, since scrub takes a snapshot anyway (and should tell you
which sstable
Ok i've run scrub on the 3 nodes and the problematic row
Error validating row
DecoratedKey(92668395684826132216160944211592988451,
353339332d3134363533393931)
The full message is
WARN [CompactionExecutor:2700] 2012-06-14 14:26:42,041
CompactionManager.java (line 582) Non-fatal error
I was able to figure out that 353339332d3134363533393931 is the row key
while no idea what is 92668395684826132216160944211592988451 part?
sstable2json also fails with validation error on this row key
now since I have lost data for this row - how do I find out that was
the root cause?
One of the column names on the row with key 353339332d3134363533393931
failed to validate with the validator for the column.
If you really are after what column is problematic, and are able to
build and run cassandra, you can add debugging info to Column.java
protected void
