Niket Vilas Bagwe created CASSANDRA-20022:
---------------------------------------------
Summary: ChannelProxy read might not read complete buffer causing
"Checksum didn't match" error for sstableloader
Key: CASSANDRA-20022
URL: https://issues.apache.org/jira/browse/CASSANDRA-20022
Project: Cassandra
Issue Type: Bug
Components: Tool/bulk load
Reporter: Niket Vilas Bagwe
CassandraCompressedStreamWriter expects the outbuffer to be filled completely
while issuing the read call for ChannelProxy. Please refer the code snippet
below from CassandraCompressedStreamWriter.
{code:java}
out.writeToChannel(bufferSupplier -> {
ByteBuffer outBuffer = bufferSupplier.get(toTransfer);
long read = fc.read(outBuffer, position);
assert read == toTransfer : String.format("could not read required number
of bytes from file to be streamed: read %d bytes, wanted %d bytes", read,
toTransfer);
outBuffer.flip();
}, limiter); {code}
This is problematic since the underlying FileChannel implementation for
ChannelProxy can have reads less than the buffer size causing the file upload
operation to fail.
Also, to catch the above problem an assertion is used instead of using an
Exception. Assertions have to enabled explicitly by passing -ea flag to the
java runtime.
If assertions are disabled, the above partial read operation can cause an
invalid buffer to get transferred to the cassandra peer causing "Checksum
didn't match" error on the peer as shown below.
{code:java}
org.apache.cassandra.streaming.StreamReceiveException: java.io.IOException:
Checksum didn't match (expected: 1967481976, actual: 2124622197)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:60)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:38)
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
at
org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:172)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Checksum didn't match (expected: 1967481976,
actual: 2124622197)
at
org.apache.cassandra.db.streaming.CompressedInputStream.maybeValidateChecksum(CompressedInputStream.java:206)
at
org.apache.cassandra.db.streaming.CompressedInputStream.loadNextChunk(CompressedInputStream.java:157)
at
org.apache.cassandra.db.streaming.CompressedInputStream.reBuffer(CompressedInputStream.java:121)
at
org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:90)
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
at
org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:62)
at
org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:93)
at
org.apache.cassandra.db.marshal.ByteArrayAccessor.read(ByteArrayAccessor.java:101)
at
org.apache.cassandra.db.marshal.ByteArrayAccessor.read(ByteArrayAccessor.java:38)
at
org.apache.cassandra.db.marshal.AbstractType.read(AbstractType.java:500)
at
org.apache.cassandra.db.marshal.AbstractType.readArray(AbstractType.java:480)
at
org.apache.cassandra.db.ClusteringPrefix$Serializer.deserializeValuesWithoutSize(ClusteringPrefix.java:441)
at
org.apache.cassandra.db.Clustering$Serializer.deserialize(Clustering.java:164)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:477)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:434)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:84)
at
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:62)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.streaming.CassandraStreamReader$StreamDeserializer.hasNext(CassandraStreamReader.java:249)
at
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:67)
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:205)
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
at
org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:107)
at
org.apache.cassandra.db.streaming.CassandraStreamReader.writePartition(CassandraStreamReader.java:175)
at
org.apache.cassandra.db.streaming.CassandraCompressedStreamReader.read(CassandraCompressedStreamReader.java:98)
at
org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
at
org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:53)
... 5 common frames omitted {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
