[jira] [Updated] (CASSANDRA-6366) Corrupt SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Jurik updated CASSANDRA-6366:
--
Environment: 1.2.10, LeveledCompaction, CQL3 (was: 1.2.10)
Corrupt SSTables
Key: CASSANDRA-6366
URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
Project: Cassandra
Issue Type: Bug
Environment: 1.2.10, LeveledCompaction, CQL3
Reporter: Matt Jurik
We ran into some corrupt sstables on one of our 8-node clusters running
1.2.10 (since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a
single node. After doing a nodetool scrub and then a nodetool -pr repair
for the cluster, we were left with 2 nodes reporting 3 corrupt sstables.
All nodes appear healthy; fsck and our raid controllers report no issues. The
sstables were written out during normal operation; there were no machine
restarts or failures anywhere near the sstable file timestamps.
Curiously, I figured out how to read all 3 of our corrupt sstables, though I
have no idea why this works. Additionally, it seems that I'm able to read all
OnDiskAtoms as specified in the row header, so the data seems intact.
{code}
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 381fdb9..8fce5f7 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
public boolean hasNext()
{
+ /*
+ * For each row where corruption is reported, it is the case that we
read more data from the preceeding row
+ * than specified by dataSize. That is, this iterator will terminate
with:
+ * inputWithTracker.getBytesRead() dataSize
+ */
return inputWithTracker.getBytesRead() dataSize;
}
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
index 1df5842..718324c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
@@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
{
try
{
-if (row != null)
-dfile.seek(finishedAt);
+// Magically read corrupt sstables...
+// if (row != null)
+// dfile.seek(finishedAt);
assert !dfile.isEOF();
// Read data header
diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java
b/src/java/org/apache/cassandra/tools/SSTableExport.java
index 05fe9f6..ed61010 100644
--- a/src/java/org/apache/cassandra/tools/SSTableExport.java
+++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
@@ -432,7 +432,7 @@ public class SSTableExport
*/
public static void export(Descriptor desc, String[] excludes) throws
IOException
{
-export(desc, System.out, excludes);
+export(desc, new PrintStream(json), excludes);
}
/**
{code}
Otherwise, I get a stacktrace such as:
{code}
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
dataSize of 72339146324312065 starting at 80476328 would be larger than file
/Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
length 109073657
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:176)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:84)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:70)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:422)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:435)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
[jira] [Updated] (CASSANDRA-6366) Corrupt SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-6366:
--
Priority: Minor (was: Major)
Scrub should be able to fix a corrupt row size like this. What does it log
when you scrub it?
Corrupt SSTables
Key: CASSANDRA-6366
URL: https://issues.apache.org/jira/browse/CASSANDRA-6366
Project: Cassandra
Issue Type: Bug
Environment: 1.2.10, LeveledCompaction, CQL3
Reporter: Matt Jurik
Priority: Minor
We ran into some corrupt sstables on one of our 8-node clusters running
1.2.10 (since upgraded to 1.2.11). Initially, we saw one corrupt sstable on a
single node. After doing a nodetool scrub and then a nodetool -pr repair
for the cluster, we were left with 2 nodes reporting 3 corrupt sstables.
All nodes appear healthy; fsck and our raid controllers report no issues. The
sstables were written out during normal operation; there were no machine
restarts or failures anywhere near the sstable file timestamps.
Curiously, I figured out how to read all 3 of our corrupt sstables, though I
have no idea why this works. Additionally, it seems that I'm able to read all
OnDiskAtoms as specified in the row header, so the data seems intact.
{code}
diff --git
a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
index 381fdb9..8fce5f7 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
@@ -180,6 +180,11 @@ public class SSTableIdentityIterator implements
ComparableSSTableIdentityIterat
public boolean hasNext()
{
+ /*
+ * For each row where corruption is reported, it is the case that we
read more data from the preceeding row
+ * than specified by dataSize. That is, this iterator will terminate
with:
+ * inputWithTracker.getBytesRead() dataSize
+ */
return inputWithTracker.getBytesRead() dataSize;
}
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
index 1df5842..718324c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java
@@ -167,8 +167,9 @@ public class SSTableScanner implements ICompactionScanner
{
try
{
-if (row != null)
-dfile.seek(finishedAt);
+// Magically read corrupt sstables...
+// if (row != null)
+// dfile.seek(finishedAt);
assert !dfile.isEOF();
// Read data header
diff --git a/src/java/org/apache/cassandra/tools/SSTableExport.java
b/src/java/org/apache/cassandra/tools/SSTableExport.java
index 05fe9f6..ed61010 100644
--- a/src/java/org/apache/cassandra/tools/SSTableExport.java
+++ b/src/java/org/apache/cassandra/tools/SSTableExport.java
@@ -432,7 +432,7 @@ public class SSTableExport
*/
public static void export(Descriptor desc, String[] excludes) throws
IOException
{
-export(desc, System.out, excludes);
+export(desc, new PrintStream(json), excludes);
}
/**
{code}
Otherwise, I get a stacktrace such as:
{code}
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
dataSize of 72339146324312065 starting at 80476328 would be larger than file
/Users/exabytes18/development/yay/corrupt-sstables/corrupt-files3/my_keyspace-my_table-ic-40693-Data.db
length 109073657
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:176)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:84)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:70)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:203)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:157)
at
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:144)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:391)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:422)
at org.apache.cassandra.tools.SSTableExport.export(SSTableExport.java:435)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:517)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
