[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15476024#comment-15476024
]
Cameron Zemek commented on CASSANDRA-12590:
-------------------------------------------
With added extra logging
{code:title=BigTableWriter.java|borderStyle=solid}
protected long beforeAppend(DecoratedKey decoratedKey)
{
assert decoratedKey != null : "Keys must not be null"; // empty keys
ARE allowed b/c of indexed column values
if (lastWrittenKey != null && lastWrittenKey.compareTo(decoratedKey) >=
0) {
ByteBuffer key = decoratedKey.getKey();
if (decoratedKey.getToken() instanceof LocalPartitioner.LocalToken)
{
LocalPartitioner.LocalToken token =
(LocalPartitioner.LocalToken) decoratedKey.getToken();
logger.warn("BigTableWriter::beforeAppend (" + (token == null ?
"null" : ByteBufferUtil.bytesToHex(token.getTokenValue())) + ", " + (key ==
null ? "null" : ByteBufferUtil.bytesToHex(key)) + ")");
} else {
logger.warn("BigTableWriter::beforeAppend " + (key == null ?
"null" : ByteBufferUtil.bytesToHex(key)));
}
throw new RuntimeException("Writing into " + getFilename() + " Last
written key " + lastWrittenKey + " >= current key " + decoratedKey);
}
return (lastWrittenKey == null) ? 0 : dataFile.position();
}
{code}
got the following
{noformat}
Sep 06 16:58:54 ip-10-222-128-182.eu-west-1.compute.internal cassandra[7338]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000007657683449634a374136764138703051562d6a48697165793065506354,
7a433144785f615a6e36417657683449634a374136764138703051562d6a48697165793065506354)
Sep 06 18:26:05 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(000000608fa527000000003578666a493271484e48505f514c574b544b486a454c555a6661316164,
4335485f61584e2d4c62593578666a493271484e48505f514c574b544b486a454c555a6661316164)
Sep 08 13:24:01 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(0000000000000000000000494b324c5268554f4c546852554143795f31634d724e385f4975797873,
634c5155767a77494a5f32494b324c5268554f4c546852554143795f31634d724e385f4975797873)
{noformat}
So first 11 bytes of the LocalToken ByteBuffer have been corrupted.
> Segfault reading secondary index
> --------------------------------
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
> Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x00007fd7354749b7 [0x00007fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x00007fd7351b35b8 [0x00007fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x00007fd736404dd8 [0x00007fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x00007fd736e09638 [0x00007fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x00007fd736e01a48 [0x00007fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x00007fd736417c1c [0x00007fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x00007fd736541e6c [0x00007fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x00007fd7357957b4 [0x00007fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x00007fd735938d8c [0x00007fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
