[
https://issues.apache.org/jira/browse/CASSANDRA-11468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263926#comment-15263926
]
Marcus Eriksson commented on CASSANDRA-11468:
---------------------------------------------
If I'm reading the code correctly it is because during normal operation we only
flush when the buffer gets full - with compression we flush 64k of uncompressed
data into a chunk on disk. This means that for the first chunk, we guarantee
that positions 0 -> 64k [are in the first
chunk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java#L230].
If we for example have force flushed a 10k first chunk and ask for position
15k, we will read the first chunk on disk, but the actual data is in the next.
Since this cannot happen during standard operation I doubt we should fix it as
it would probably force us to rewrite the compression chunk handling (unless
anyone has a brilliant idea how to fix it safely)
> Reading from early opened, and compressed, sstable throws
> CorruptSSTableException
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-11468
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11468
> Project: Cassandra
> Issue Type: Bug
> Reporter: Blake Eggleston
> Assignee: Marcus Eriksson
> Fix For: 3.0.x
>
>
> When a compressed sstable is early opened during compaction, reading from it
> can fail with the following exception:
> {code}
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> build/test/cassandra/data/ks_1459378971131/early_open_test-893eb3d0f6cb11e59e1b4f343b985d3e/ma-3-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:130)
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:46)
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:36)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62)
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:715)
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:482)
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:459)
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
> at
> org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:363)
> at
> org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:393)
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.executeInternal(SinglePartitionReadCommand.java:950)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:397)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:76)
> at
> org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
> at
> org.apache.cassandra.io.sstable.EarlyOpenTest.earlyOpen(EarlyOpenTest.java:160)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: java.io.EOFException
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readShort(RebufferingInputStream.java:164)
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedShort(RebufferingInputStream.java:170)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:366)
> at
> org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferUtil.java:392)
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:97)
> ... 41 more
> {code}
> In my testing, it's been the second read that fails, and is because the
> buffer the sstable iterator is trying to read from has it's position set to
> it's length. Uncompressed sstables work fine.
> I've pushed up a branch with a reproducing unit test
> [here|https://github.com/bdeggleston/cassandra/tree/11468]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
